I'm probably explaining poorly. Thats my experience most of the time when the hitch is loaded with a trailer it doesn't wobble anymore and doesn't make much noise. On my XC90 it still does though lots of it. Its very loud over every bump and imperfection in the road.
Few people recommended a few methods to fix it. I just ordered this pin thing thats supposedly removes all the slop. We'll see. Just didn't know if any one else had run into this.