Painting - Exterior paint scrapping

Is it better to scrape off as little paint as possible when getting ready to do exterior trim painting or work at it to get off as much paint as possible? My window and garage door trim on the south and west sides of the house have parts that could be peeled off with your fingers and if I work at it, the rest of the paint could be scraped off. But do I want to do that much scrapping or is it better to just take off the stuff that comes off easily and paint over the rest?

To get ready for painting, I would pressure wash the area first and scrape off all the remaining loose paint. Sand to blend in the paint with the bare areas. Give it a good primer coat before painting. I prefer oil primer.