But I will still have recirculating air from inside though right?
If the "defroster" mode is valuable to you in your climate, you may want to take a second look at blocking things off also? I did something similar on a brand X car and driving in the rain or very moist conditioned required a towel on the seat next to me for wiping purposes, especially at night. Might not be the case in your situation, but keep all the scenarios you'll encounter in mind.
What might be cool is to make a flat panel like you describe but hinge it and put a vacuum motor or electric actuator on it to open up on fresh air settings....