Fireplaces, Heating Stoves, Flues and Chimneys - Fireplace very drafty

We just moved into our house and I have noticed a huge draft from the gas direct vent fireplace. I removed the bottom grill where all the controls, gas line and blower motor are located. On one side there is a large opening for a small gas line and on the other side there is a hole for the electic wiring. There is a huge amount of cold air flowing into the house through these openings and also where the face of the fireplace meets the fireplace trim molding. The fireplace is flush with the inside wall and sits inside a box framed from a two foot overhang and the exhaust is straight out the back of the fireplace.
I was thinking of sealing the holes in the fireplace box with expanding foam, caulking around the trim molding, pulling off the vent pipe and sealing with stove gasket cement. Also I removed the soffit below the fireplace overhang and there is no insulation but I have no idea what type of insulation to use if any.
Any advise, the draft is very noticeable in the living room. We currently have the lower grill cover with blankets but this kind of takes away from the look of the fireplace.

Today's sealed up homes must have fresh air intake for combustion, as using the air in your home, well, it is depletable. The triple wall flue pipe uses the outer ring to allow intake of fresh air into the box. It appears you have poorly insulated flue or it is not sealed properly to the top of the firebox, or where the smoke exits. Is there any way to get to the innards?

It is a heatllator unit with a horizontal termination. I can take off the front glass and reach up into the flu on the top rear of the firebox where the exhaust exits and touch the exhaust box (not sure on the terminology but I mean the aluminum box you can see on the outside of the house). I know that this pipe vents directly to the outside so it surprised me I didn't feel much of a draft. After your response I think it is because the exhause box is not sealed where it meets the house sheathing. My home is a walkout so the only access I have is to climb a ladder and remove the exhaust box. I plan on doing this over the weekend and see what I can find.