Fireplace down draft

[Dutch7]

I have a wood burning fireplace with a gas-assisted igniter. It does have glass doors.

When it is cold outside, I can feel the down draft come through the air slots around the glass doors! Sometimes when it is cold enough outside, ice might form! (Even around the gas igniter key slot!) I have made sure the damper is closed all the way.

When it is cold outside and I wat to start a fire, I have to be very carefull, because when I open the damper, the cold air really rushes down and it is very hard to start the fire, even with the gas igniter! Once the fire is going, it isn't an issue. But once the fire dies, and the flue get cool, the cold air comes down again.

Any ideas?

Thanks!