Correct. Both the ball mount and the phone mount have 1" ball connectors, so you need the connecting arm to connect them (which also allows the mount to pivot).
I have the RAM X-grip on my bike which works great, but I also have the self leveling cup holder, so it's very easy to change between them using the same mount as I just loosen the middle connecting rod and swap the X-grip out for the cup holder as it also has a 1" diameter ball connector.