My husband has this a little bit. He also didn't start driving until he was almost 30 because he grew up in NYC, didn't need a car in college, and then after we lived in cities where we didn't need one for many years. It wasn't until we moved to the suburbs after having a child that he needed to be able to drive. He's much better now that he's been driving a few years, but he still like to plan ahead if he's going somewhere unfamiliar. He'll check the route on google and plan which roads he'll take depending on the location. He doesn't like having to make left turns over busy roads, for example, so he'll try to plan around that. The planning helps him avoid the anxiety once he's driving.

Maybe for this you could plan out the route ahead of time. If you're nervous about your child having a reaction in the car, look in advance if there are hospitals or other places along the way you would be able to stop and know how you'd get to them. I think with a fear like this, exposure is the best therapy, so it's good to try your best. But, it's also not unreasonable to tell your husband you're already nervous about the appointment itself and you'd really appreciate him being there with you for support even if that just means driving and waiting in the car.