All of the above with just one more suggestion, it would be a good idea to replace all of the antifreeze. It's a great time to do the thermostat and new antifreeze, you'll be ready for this summer. Don't forget to check your hoses.
Also, while you have the heater box out, replace any foam that you can with new stuff. It is worth the time to get it all done correctly. You can find some replacement foam at home improvement/hardware stores. (especially the foam around the heater core)