I'm currently on an airplane, using mosh to connect to several servers back on the ground.
I originally started the sessions while on my tethered mobile phone connection and as soon as I connected to the in-flight WiFi the sessions reconnected seamlessly. Even though the links have more than 800 ms (!) of latency, the sessions feel responsive due to the local echo. Plus, if I send a ^C
, it actually works :D
I've been using this now for around a year exclusively. It's one of the best pieces of software I've had the pleasure of using.
The research paper and video found here are fascinating!
The only caveat is that there is no scrollback like a typical xterm. You will need to use a tmux session (or similar) on the server to get scrollback. I haven't found this to be too much of a burden.
anything's legal if it can't be traced back to him
there are non-american cloud providers
mosh is great for ameliorating latency if he's developing on the opposite side of the world
monero is probably the least traceable currency, although it's hard to acquire without going through an exchange
if he wants to do really sketchy searches / setups public libraries are his friend
make sure he's using a VPN he's vetted, TOR has some problems
i hear botnet access is cheap if he wants to run a scraper
he should make sure any code he's developing locally is secured with strong encryption and stored in a few different places
> I use tmux when I ssh into machines to preserve remote sessions in the event I get dc'd for whatever reason
You might be interested in mosh instead of ssh. It's UDP-based and persists across disconnects and even network changes. Makes it great for mobile devices like tablets and laptops, networks with bad connectivity, and thanks to instant feedback on typing, it makes high-latency links more tolerable.
Still need tmux
, dtach
, or screen
to save your session between local reboots, though. I use tmux and mosh together: mosh for disconnect-proofing and tmux for "windowing", multiple views of a single session, and reboot-proofing. Not having to reconnect constantly, especially on mobile devices, is great.
It sounds like the real issue is you're leaving the server attached to your pty. Install GNU Screen or tmux and run the server in one of those, then you don't need to stay connected.
Alternatively, try mosh.
Advice for anyone out there trying to use SSH on tethering or other flaky internet connections: Give Mosh a try. It handles lag and packet loss much better than SSH. I've been on connections where webpages would not load at all but my existing Mosh session was still usable. As a bonus, when you switch Internet connections or resume your laptop from suspend, it automatically reconnects within 3 seconds. I was a contributor to Mosh some years back so feel free to ask me questions about how it works.
Now, a rant: Caltrain started working on installing WiFi back in I think 2007. The current claim is that it'll be ready in 2022 with the new electric trains. We'll see if that happens. 15 years...
> tmux over ssh is the real reason for using tmux (or screen for that matter)
Sir, if you'd give me a moment of your time, I'd like to introduce you to the good word of our lord and saviour, mosh.
Joking aside, mosh is great for laggy networks, intermittent connections, and mobile devices that hop networks. Not having to reconnect constantly is awesome and it has nearly completely replaced normal ssh for me. (Only thing it doesn't do is port forwards, so I still have to use ssh for those.)
Won't replace tmux, but it pairs better with it than ssh does.
Is that built on TCP or UDP ?
Mosh's ability to keep the connection alive when roaming or putting your computer in sleep mode thanks to UDP is really a killer feature. Any SSH replacement should have this IMHO.
My friend... https://mosh.org
> SSH waits for the server's reply before showing you your own typing. That can make for a lousy user interface. Mosh is different: it gives an instant response to typing, deleting, and line editing. It does this adaptively and works even in full-screen programs like emacs and vim. On a bad connection, outstanding predictions are underlined so you won't be misled.
I might be mistaken, but I think the USB and ethernet share bandwidth on the rpi. An SSH session can sometimes feel laggy regardless though. One thing you can try is Mosh (https://mosh.org/). It makes terminal sessions feel more responsive, especially if you're out and about.
Not quite clear from your question though; is file access itself slow?
I've had to do stuff on a server on such a connection via ssh. That's where you start to praise mosh. The guy had two separate satellite connections for two sites across the street so anything between those two sites had double that latency. You can't imagine the sanity toll it takes.
Thanks. Apart from being much more simple setup, it also made SSH sessions more responsive, as instead of going first to intermediate server and then to target machine, traffic now goes straight to target machine via best route possible. I also recommend Mosh, which is using ssh under the hood, but makes it better :)
I’ve been using Blink for some time and highly recommend it. It is expensive, but you can build and install it yourself if you’re that way inclined (I decided to buy it primarily to support the developers).
It’s full of really useful features, but the most valuable to me is support for Mosh. Incredibly useful if you use your iPad in locations where connectivity is poor (in my case, using a cellular connection on a train). The reference Mosh implementation is GPLv3, which makes it difficult for developers to integrate it into non-GPL apps. I contacted Panic to ask if they’d implement Mosh support in Prompt and they said they wouldn’t be able to until a differently licensed implementation was available.
I'm using tmux exactly for its rich pane and window management. It's great for e.g. making ad-hoc dashboards for debugging sessions that survive when I move my laptop from one network to another.
My basic remote workflow now involves mosh+tmux for basically everything for this reason.
> Very nice! Emacs on a tablet might be the killer feature that converts me to your workflow.
Don't. You can get the best of both worlds by using emacs-daemon and just leaving it running. When I'm at my PC I use the normal GUI, and when I'm away from it I connect to it using mosh and fire up a terminal instance with emacsclient. Aside from some minor syntax highlighting issues that I squash when I notice them, GUI and terminal look and function similarly enough that I can swap between them with little issue.
Instead of trying to do everything in the terminal, focus on getting the terminal and GUI configurations to look as close as possible to each other so that which one you're in matters less. That way you can choose which set of pros/cons you need at a given time instead of limiting yourself unnecessarily.
I would look at this at an architectural, and protocol by protocol basis.
WAN optimizers are a big inefficient hammer and a last resort.
For example, deploying RDP v8+ and SMB3+ on a windows network would outperform the best optimizer with their legacy counterparts, because the protocols themselves are latency tolerant, and leverage compression and deduplication at the application level.
Similarly, for an IOT type deployment I would probably just want a reverse proxy like NGINX on site for caching updates and configuration repositories on demand, a little local DNS or routing magic for fault tolerance, and a jump box that I can run screen on in case I get dropped. Or a shell like https://mosh.org/
For basically everything, you'll want to look at UDP modes if available. Stateless but application specific usually trumps on high latency high loss links. OpenVPN has a UDP encapsulation mode.
You should be able to encrypt just the payload as well in many VPN suites. Not as secure, but better than running your traffic raw while still giving you the satellite fix-ups. I occasionally do this with Cisco DMVPN on low MTU links to reduce overhead. DMVPN + EIGRP is fairly scalable FWIW.
Have you tried mosh to mitigate against the spotty connection? It is fantastic – I could not work remotely without it: https://mosh.org
Even with a solid but mid-to-high latency connection it helps by minimizing the annoyance of not getting feedback each keypress by having the client-side terminal display updates immediately even if it is waiting on the server.
JuiceSSH has a bunch of features that ConnectBot didn't have the last time I used it. It's been a while so maybe ConnectBot has added features in the meanwhile.
The big one for me is MOSH support. JuiceSSH also has modifier keys built into it. With ConnectBot, I instead had to install a keyboard (Hacker's Keyboard?) that had those keys. JuiceSSH has forwarding and SOCKS support.
I'll concede a point that JuiceSSH is closed source and commercial, while ConnectBot is free and open source. I've found that for how rarely I need an SSH client on my phone, I need the features that JuiceSSH offers.
Check out MOSH - it can help you with other connection related terminal issues. It uses SSH as a backend for authentication and it doesn't need any system-wide modifications, so it can be used on user accounts without any extra privileges.
> Typing over an SSH connection to a slow remote machine is an exercise in frustration.
Try mosh instead of ssh, you'll love it. It does the prediction echo in a way that doesn't ruin ncurses apps and, due to using UDP, remains active through loss of connectivity which is useful for bad connections or mobile devices that roam. Negotiates the connection over ssh then immediately terminates it once the session is established.
The main drawback is you lose your terminal emulator's scrollback so you have to be more diligent about using less
or use something like termux to provide it. The benefits greatly outweigh that issue, though.
Edit: just noticed someone else suggested it already. Oh well, leaving my comment up anyhow since I added some detail.
> Get rid of network lag.
> SSH waits for the server's reply before showing you your own typing. That can make for a lousy user interface. Mosh is different: it gives an instant response to typing, deleting, and line editing. It does this adaptively and works even in full-screen programs like emacs and vim. On a bad connection, outstanding predictions are underlined so you won't be misled.
> Hopefully this works on mobile. I'd love to be able to test random script ideas I get while out and about.
This may not be useful depending on what kind of scripts you're making, but one way to do this is to install the headless Godot and use it to run a script. Basically you just make a script that extends SceneTree or MainLoop and run it with godot -s script.gd
. Even works with Mono "scripts" (C#, or in my case F#), though in that case you have to run godot --build-solutions
first to recompile.
You can set this up on a system you have remote access to (via ssh or mosh) and some things remotely. For example, I've been doing that on a Chromebook with mosh installed to experiment with using F# in Godot land; I've just been trying to get accustomed to how Godot and F# do things and work together, so I don't need the full editor, just a way to test little things, and it's worked pretty well for it.
mosh. It makes using SSH over poor-quality connections slightly more usable, and the "roaming" feature is a killer: you can sleep your laptop at home or work, wake it back up on a coffee shop's Wi-Fi and your SSH connection is still active.
Some cool things for your to try:
-setup ssh keys
-disable password authentication so that ssh can only be used with keys
-Try Mosh! (this one I'm still playing around with myself!)
mosh
is a shell, you run it inside of another terminal. PuTTY
is a standalone application (since OP stated he uses Windows) for SSH access to other systems. You can't simply replace one with the other because they are different things.
While there are portable versions of mosh
that can run on various platforms, but the Windows version is only available as a Chrome extension and we can't even be sure OP uses Chrome
> also if your ip changed the session would drop. How to maintain a session if the user keeps changing where the packets go. Sounds fucky
Depends on the session and how the server wants to handle it.
If it's a session on a webserver, then the session won't be deleted unless the server specifically tracks the remote address and drops the session because of it.
Web Sessions work via a key and a chunk of saved data on the server that the key points to. The key being stored as a cookie in the browser.
If it's an SSH session, it's possible to still tunnel through the Tor service. I'm pretty sure the way it's handled there is that it just closes the old connection and starts a new one real quick. Could look into mosh with regards to that.
Yes, but it's not a naïve local echo, it uses an algorithm the authors called "speculative local echo" which adjusts its behavior if the authoritative screen state eventually received from the server doesn't match the local echo. You can read a high level description of it in this paper: https://mosh.org/mosh-paper.pdf
I've recently changed from an ssh reverse tunnel to using frp[https://github.com/fatedier/frp] in production ssh is arguably the greatest piece of software on the planet, but it's not design for long running jobs. If your connection to your ISP is bounced the tunnel will lock up on the vps side. Trying to keep the connection alive through the bounce is flaky (refer TCPKeepAlive et.al. for more) This problem is solved for interactive sessions with with mosh[https://mosh.org/] but this is not a solution for tunelling your connection. What you are trying to do is very doable and it offers an awesome solution for self hosting, everyone should be doing this IMHO but frp written in go gives you more options. Try it YMMV It also has a funky dashboard which reports the status and settings of your connections
Mosh will work better on the iPad than regular ssh. It can automatically reconnect, which is helpful because the iPad will quit your terminal client when it is in the background too long. Look at running in tmux or screen (I prefer tmux :-) so you can jump between your desktop and iPad without losing your place.
After your get Mosh installed, the next thing you need is a terminal client. It's $20, but Blink is fantastic.
This is the set up I use all the time for writing and programming from my iPad mini. I do have JK set up to escape from normal mode, but everything else works well.
If you're adamant about wanting a "shell login" type solution, after taking into account all the other feedback from this thread, I suspect you'd be much better served by something like Mosh than ssh.
You can see which shell you are using with the ps -p $$
command. Make sure it really is bash. If not, make sure you have bash installed and you can change your user's default shell with chsh
.
For interactive sessions you should make sure you are using the closest AWS region to where you connect from. The larger the distance the higher the latency. If the latency is still too high, try Mosh.
What are you going to run on your EC2 workstation? There may be easier options than using EC2 directly. Cloud9, for example, is a browser based IDE that runs on top of EC2. It automatically starts and stops your instance so you only pay for when you're using it. If you're going to run any graphical applications, AWS Workspaces has a Linux option that also starts and stops automatically to save you money.
Thanks for mentioning that. I've heard about it before but forgot about it. The way it works sounds a lot like how mosh works (have own client and server, use ssh for negotiation) which is a cool idea for how to make it happen.
No sign of it in Debian still though, not even in sid, so it's still off my radar as far as "Wayland is viable for me" is concerned. I use Debian because I expect the base system to be boring, solid, and available through the package manager. I'll pull some stuff from external sources, but not things I expect to work and stay working and be updated reliably for security issues. :/
In a couple years when Debian's next stable happens I'll give it another look same way I've done the past couple. Maybe it'll finally be time by then.
It occurred to me that I could try to help you set it up directly if you get desperate. Since you have a Pi you could set up a temporary account and DM me the connection info, and I could attempt to set up a bare-bones emacs+merlin configuration on the account that you could use as reference or a baseline. Or if you have a cheap VPS I could do it on that, same result.
I understand if you don't want that since I'm just some random person on the internet but I figured I'd offer anyway. I've been helping friends with Linux server stuff like that for ages so I'm comfortable doing stuff like that and since emacs works on the terminal as well it'd be easy enough. Would just need ssh access and maybe mosh + tmux if the connection latency is high.
I think you can figure it out yourself eventually (maybe try a second account with a fresh configuration of everything and experiment), but figured I'd offer.
They have instructions on the site. Judging by the ./configure
make
make install
it's probably autoconf and you probably only need to adjust a PREFIX
variable or something. try ./configure --help
to see the options.
edit: the instructions are here
Do you need znc to be running in termux or do you just want znc to be running somewhere?
This is what I do for IRC:
Sign up for an "always free" GCE f1-micro VM instance.
Run irssi inside a tmux (not termux) instance on Debian Stable (or your server distro of choice) on it.
Use termux (with fish shell) and remotely mosh into the instance and reattach using the command mosh username@hostname -- tmux attach
You may choose to substitute znc or weechat in place of irssi and screen in place of tmux if that's what you're more comfortable with. The key point is the VM doesn't cost money, but you'll have to set it up yourself. Either way, I think this should be a "viable replacement" according to what you've mentioned so far.
Alternatively, I think a Raspberry Pi is also a good option (assuming you have a reliable-enough internet connection).
I don't see the benefit of running the znc bouncer inside termux itself given these other options above.
I'll just say: Even when it's going well, you never want to try to do ops from an airplane on an intercontinental flight. Your ping is measured in tens of seconds. Page loads just kinda tend to time out half the time, and you'll be raging at the hardcoded timeouts on your own dashboards. If you weren't already using mosh, even ssh is kind of a lost cause. About all you're useful for is chat. And all the while, you know you have no one to blame but yourself -- the fact that you'd even consider trying to help suggests an insane lack of planning.
Thank fuck everyone else had their shit together for that launch. I've never been so grateful for a 3G connection as when that plane touched down.
That's actually not my most stressful story, but it's probably the most stressful one that I can anonymize properly.
I do some. $work doesn't allow source code to be on laptops, so have to use devices on the Corp network.
Also a big plug for Mosh which makes uncertain network conditions much easier to deal with.
> The benefits of mobility really shined when Covid hit.
I think desktops are more productive as well as being flexible, but the Disaster Recovery utility of laptops is rather undeniable.
What you want to avoid is all of those laptops sitting in the office when the office is flooded, or knocked over in an earthquake, or rioters break in and smash everything they don't steal. This didn't happen with COVID-19 because the timeline was so relaxed, but it should be concerning to you. Thus, I decided that laptops should mostly travel with the user.
This had a few different results. One was to prioritize laptops of a size that users wouldn't be reluctant to cart around. We'd previously had issues with small-framed people lugging the likes of Thinkpad Z60m models and similar. We issued multiple power supplies (isn't USB-C the bee's knees?) as a result of operational experience with mobility.
Another result was that anyone who couldn't or wouldn't move their machine, would get a desktop. I tend to fall into that category. Those with long-lived SSH sessions who aren't using tmux
or mosh might fall into that category. They might also get a laptop, tablet, or smartphone, in addition to the desktop.
I used to do something similar. I set up a Raspberry Pi with tintin on it. I would just shell into it when I was home and mud that way, but when I was out on my phone, I would mosh into it and play from the same client. This is really nice if logging is important to you, because you only have one log to deal with, its not split between devices.
What is your location and where are your cloud servers? What are your ping times? I'm in London and I often use US-based instances and they are not really distinguishable from a server on my own LAN, while using SSH. When I started over 10 years ago I played around with instances as far away as Tokyo and the connections would have only got faster since then.
If you have SSH connections dropping, I highly recommend using mosh. I use it in laptops and my iPad, where I can even close the lid and reopen it hours later and still maintain the connection. You will not the way certain things redraw the screen is a little different when using mosh, but if you use tmux within it, it's the same as straight SSH. You only need to use it on your client and your bastion host and then use SSH from there.
This happens to me all the time, but I don't think it is neovim related.
If you have an unstable internet connection then ssh is prone to do this. Even brief interruptions of connectivity will break the connection. When the connection is broken like this ssh does not even generate warnings or error messages or anything, and there is no good way to even know for sure that this has happened. Ssh also does not allow you to roam from one connection to another, and if you do, you will experience this exact kind of breakage immediately (again, no error messages or anything).
There is an alternative to ssh called mosh, which is supposed to fix these issues.
Depends on what you're hosting, but for the most part it's probably fine. Most things aren't particularly latency sensitive
I run a lot of my testing stuff with Hetzner because it's decent specifications for the price, the worst thing about is SSH is clunky to use.
Mosh helps with the latency a lot:
Another tool a lot of people don't seem to be familiar with is [mosh]( https://mosh.org/ ) (MObile SHell). It's similar to SSH, but designed for use with for unstable connections. It seamlessly reconnects to the remote server in the event of a disconnection.
Mosh initiates its session using ssh, so you can dispense with passwords completely and use public key authentication as described in the section “Using key-based authentication” in Securing your Raspberry Pi. As for the mosh protocol itself, see the FAQ - it does have some security advantages such as running in user space and authentication of all packets.
You can also do this with a Tor hidden service.
If you're using ssh over slow/bad network links, you might benefit from mosh.
Just to throw out a couple of alternatives, in case scp://
is not satisfactory:
ssh
access, and is more optimized for bad network connections. So you could try running Vim in Mosh to work remotely.rsync
is an option. You could use it to clone your remote directories locally, and can e.g. setup a Makefile
to push your changes to the server easily.Then throw in SSH sessions that keep timing out (for remote sysadmins, there's a solution for this called Mosh) and the aforementioned file transfer issues (because most ISPs could give a rat's ass about proper upload speeds) and then you realize how far off we are from a truly ideal remote working experience.
Yes, you could coax this into working, basically using SMTP/IMAP as a transport for commands.
But don't do this.
Use mosh instead, a solution to the specific problem as you describe it.
Never heard of mosh, so if anyone else is wondering
> Remote terminal application that allows roaming, supports intermittent connectivity, and provides intelligent local echo and line editing of user keystrokes.
Mosh is a replacement for interactive SSH terminals. It's more robust and responsive, especially over Wi-Fi, cellular, and long-distance links.
Mosh is free software, available for GNU/Linux, BSD, macOS, Solaris, Android, Chrome, and iOS.
autossh is probably better if you want a tunnels set up (the topic of the post) but in case you haven't seen it yet, just for interactive shell logins that you want to always stay connected https://mosh.org/ is far nicer.
Also, check out Mosh. It works great on the Chromebook, and provides a great mobile environment. Mosh works on lowering latency as much as possible, and can automatically re-connect. Close the Chromebook, move to a different network, open it back up and you are right where you left off. Add tmux for one of the best terminal environments you can get!
Think SSH but all the annoying problems with SSH are resolved. e.g losing the session when your laptop sleeps or loses connectivity, unresponsive typing on poor connections.
The website sums it up pretty well: https://mosh.org
Yes! It's a credibility hit. If he's wrong about that subject I happen to understand, how can I know he's not wrong about the others I don't know about?
I used to live in Seoul. Speedtests gave me a 0-2 ms ping and a humongous bandwidth. Because they go through optical fiber to servers located a few km away from me in the same city. Ping to my work in europe though? 350-400ms. I hated working remotely from South Korea.
Anyways, if someone wants to actually see what would be the latency to the Philippines like: https://imgur.com/fkx6APb
That's server to server, so usually a best case scenario and you'd probably need to add to that like 50ms to get to your router.
Edit: I'd like to add that latency is very important for my type of work (software developer, by the way mosh is a godsend). Other kind of remote works can be less impacted.
Thanks for the details! It's an interesting idea for sure.
Depending on what languages you develop in, and your preferred editor/environment setup, you might benefit from looking into setting up a server you can ssh (Mosh actually) into. It'll give you access to a lot more power, especially useful if you're in need of compiling or whatnot. If you can wrap your brain around VIM that might make the setup absolutely perfect (along with screen or something like that, for being able to resume your session whenever), although it's also possible to use Atom with something like Nuclide's remote development feature or remote-sync or so.
Your brief description of Aspera's protocol reminded me of mosh, which is a very different application with a different purpose but achieved in a remarkably similar way.
It's a remote shell application which uses SSH for user authentication and server setup, but will then open a UDP port on which its own protocol (which they call State Synchronization Protocol) is made available. The mosh terminal itself uses SSP to provide an interactive shell experience.
I do wonder if SSP could be massaged into performing large data transfers. It's not its original purpose, but from watching their technical video it seems like a lot of the necessary parts are already present.
You can also do that with mosh although I acknowledge that mosh and tmux only overlap in terms of what users can do using either, while having each a distinct set of features and use-cases. Put plainly, they do a bit different things, but some use-cases can be solved using either, effective "roaming" with a mobila client being one of them.
Other folks have covered the basics, so here are some less common ones.
Postfix or OpenSMTPd + Dovecot > Sendmail + Courier > UUCP > Fidonet > SMTP via IPoAC > smoke signals > horseback mail delivery > MS Exchange.
I cannot overstate how BAD MS Exchange and MS Outlook are at email. Office 365 is gaining popularity largely because on-premises Exchange sucks so hard. Outlook is the only mail client I've ever heard of where profile corruption is a common problem, probably because it's the only one that uses some craptastic proprietary "PST" format inherited from the 1990s instead of mbox, maildir, or SQLite. The one and only reason it continues to hold market share is that people don't know CalDAV daemons and mail client integration (Thunderbird+Lightning, not so very frightening, indeed) exist.
Haiku OS > BeOS R5
AROS (now with SMP on amd64!!!!!) > AmigaOS 4 / MorphOS
Hm, you might try Mosh:
It allows the connection to be maintained automatically even if your home connection is interrupted and comes back up with a new IP address.
EDIT: I'm a liar, apparently port-forwarding is not supported over Mosh. You might be able to do something like call SSH from a shell script in an infinite loop, so it is restarted if it dies?
With three hops, it's definitely noticeable. Like 1-2 seconds, but it's usable and I generally use it to run one or two commands at a time. Reducing the hop count makes it pretty much similar to normal ssh, then I do destination whitelists for authorized devices(with the normal ssh key auth. I'm working on a way to do PAM based 2fa for adding ssh keys next, but sending the auth codes to a device over i2p. I just do this for fun because I think it's neat). I've been trying to figure out how to make mosh work and I'm going to try increasing the hops again. Back when I started doing it I did the ssh-over-tox for the practical paranoid which was about like one hop. ssh-over-tor to a hidden service just feels marginally faster than 3 hops over i2p, but I don't know for sure that it was and it doesn't have i2p's ability to whitelist by destination. But reducing the hops was a big deal.
Thanks for responding!