Get a belt sander. Super handy for stuff like this :)
I'd say a hand held one is pretty versatile, and not too expensive. Here's a random one from Amazon to show what I'm talking about.