Yes, something like that is needed. Also you would have to send xauth keys and import them on the other side. I was hoping for already existing solution. It make me google once more if there are any solutions, and I found xpra (http://xpra.org/). It is said to be "screen for X11". Basically you run X server on remote host and connect to it from various clients. Like screen. I'll give it a go.