This app was mentioned in 4 comments, with an average of 1.00 upvote
You don't mention baseband etc. I'm on the latest version with the latest nightly 02AUG.
I have this compass app: -
https://play.google.com/store/apps/details?id=com.basicapp.gl_compass
When loaded it needed calibration and I followed the instructions. I had to do it a fair bit more than it suggested but it calibrated OK.
http://imgur.com/gallery/MwH8m
I have never noticed this being a problem.
I flashed sultan mod again and (i think the reason it works now) installed an app called LED Compass (here) that helped me recalibrate my GPS properly.
Well the Google instructions say to do it while maps is open. However I did it while I had LED Compass open and I watched it straighten itself out after I did one figure 8 movement.
to recalibrate I use this app:
https://play.google.com/store/apps/details?id=com.basicapp.gl_compass
it works for me.