Painting - Touchup sanding caulk

I put in a large amount of new trim and in some places I have some excess caulk on the trim.

Is the best way to get rid of this with a 320 grit or so sanding sponge? Generally I won't have to hit the actual caulk line that I want to keep as I think sanding that even with 320 may ruin it.

I had one person tell me to rub it off with my finger but if I'm near the actual caulk line it may damage it. I primed with oil based primer before caulking and sanded to 320 so I don't want to do a lot of damage.

BTW, I used the new GE XST 100% silicone paintable caulk. Only drawback is that it's $4 a tube.

In the same circumstances, I would rub off as much excess as possible, then sand lightly once cured.

Caulking isn't really sandable. Rubbing the excess with a rag might be easier on your finger. Get it the best you can, paint and see what it looks like. Sometimes sanding the paint and what caulking is exposed will work better than trying to sand just the caulking..... and then apply another coat of paint.

Thanks all,

I'm just being very particular. I installed 15 or so window jambs/casing with stools and I under estimated the amount of work by about a factor of 10. Actually making the jambs + casing was not that hard but after you add up filling nail holes, sanding, priming, sanding, caulking, top coat 1, sanding, top coat 2 ... well you get the picture. A lot of the caulking is along a wood to wood joint which is more critical than the typical wood to drywall joint with base or crown.

If you live in a house with trimmed window openings consider yourself lucky!