Doors and Windows - sliding glass patio door roller removal

I am trying to figure out how to remove and replace the rollers on a sliding glass patio door that probably dates to the early 1960's. The roller is attached to the bottom of the on a bolt that serves as its axle. The bolt is flat-headed and flush with the door surface on one side and fastened to the other side by a nut on the threaded end. The end of the threaded side has a slot into which you place a screwdriver to hold the bolt in place while loosening the nut for removal. I have done this. Then I attempt to tap the bolt out to remove it, but it doesn't budge with medium force. I am afraid to hit it really hard and risk breaking the glass.

There seems to be some kind of keeper that holds it in place but I can see nothing like one when looking in at the roller. Is anyone familiar with this kind of bolt / axle? Dop I have to drill or hacksaw this thing out? Any suggestionm would be helpful.