Walls and Ceilings - Removing bat & board on interior walls

The people we bought our house from closed in a carport and instead of removing the bat & board siding from the 2 former exterior walls, they decided to finish the other 2 walls with bat & board as well. Crazy. We are in the process of removing the siding so we can redo the walls. We have the majority of the small strips off the wall with only a few left. The larger planks are another issue. I'm guessing we should remove these as well so the walls will be flush and even, correct? If so, we have lots of hard work ahead of us. These people loved using nails and used 3-4" screw tail nails.

You can hang drywall or paneling over the siding but it may lay flatter if you remove it all. A lot depends on how even or level the remaining siding is.