Yeah, that's what I have, too, but you can buy this (or a million others like it--I have a super cheap alternative that works fine with my X-grip, which is exactly the kind you have), and it helps immensely at keeping your phone from flying away!
I'd recommend an x-grip with one of the rubber tethers and a good phone case. I've got an otterbox for my s5 and it's worked out great so far.
You shouldn't have much trouble as long as you have a case on your phone and you're moving, but if you're worried about it (and your phone has wireless charging) you can try this
If that's an x-grip mount, they make the rubber band retention strap for the tablets too. Holds it rock solid.
This thing.