you look into jack plates at all?
can't say I've installed one, but getting a solid aluminum mount seems like it could help.
https://www.amazon.com/T-H-Marine-65012-Manual-Plates/dp/B004MZ3588
For $300 that seems like a fairly easy "good fix"
Having to keep weights in your boat sounds sketchy too
Considering using these for two of the castors: https://www.amazon.com/MaxxHaul-70148-Trailer-Wheel-1000-Capacity/dp/B008CE0FQ0/ref=sr_1_6?dchild=1&keywords=trailer%2Bjack%2Bwheel&qid=1605205299&sr=8-6&th=1