Rugs, Carpets and Carpeting - Removing left-over carpet foam from plywood

We recently purchased a 1961 cape with ancient wall-to-wall carpeting on the second floor. Once we removed the carpet, we found underneath was (what seems to be anyways) carpet foam still glued to the plywood subfloor. The problem is that this 'foam' has deteriorated so much that it looks and feels like a thick layer of sand glued to the plywood. My question is, how do we clean this up and recover the plywood subfloor? If this is not the most efficient option, should I remove/replace the plywood, or install new plywood on top of the old? Any help would be greatly appreciated!!
Thanks.

Scraper and lots of patience will remove stuck down padding. Depending on what type of floor covering you are putting down, you may want to do some sanding.