I run screen inside my mosh sessions which provides scrollback, among other features. (Tmux is another option.) ET is also a great solution, but I happen to need screen or something like it for its other features anyway.
I've been using mosh since it was announced, but I periodically try Eternal Terminal.
However I havent ever managed to get ET running on Amazon Linux which all my AWS EC2 instances run. I've tried compiling, docker, app images and nothing has worked.
Just checked this out, and the scrollback feature looks great.
I'd like to try this out, but am I understanding correctly that this needs to be installed on the server side as well as the client? That seems like a fairly large limitation compared to mosh.
Eternal Terminal collects crashes and errors in order to help us improve your experience.
The data collected is anonymous.
You can opt-out of telemetry by setting the environment variable ET_NO_TELEMETRY to any non-empty value.
Mosh has been sort of semi-neglected in a while - they did merge PRs, but did not release stable in years - but very recently, they started releasing new versions again.
I have been using mosh on and off, and actually appreciated that the software was so robust that it just works without frequent updates, and as far as I understand, most security is in ssh, so there would not really be a need for a new version.
Your understanding is sort of mistaken. Mosh does an initial handshake over ssh, but after that it’s a custom UDP protocol. There have been and could be more security bugs in that UDP protocol.
The only mosh CVE [1] was in the terminal emulator (a DoS that could only be triggered by a local user), not in the protocol. There have been no vulnerabilities in mosh's UDP protocol.
On Wireguard's homepage[1] it says "[Wireguard] is even capable of roaming between IP addresses, just like Mosh. There is no need to manage connections, be concerned about state, manage daemons, or worry about what's under the hood."
Has anyone been using Wireguard as a successful Mosh replacement? Would love to hear your experience.
It kinda works, but mosh reconnects on failure regardless of how long time has passed. Since WG is a layer below, the TCP connection eventually gets reset. It is a huge difference. Also mosh works around lag by local echo, no such thing in WG.
If I'm expecting echo, it does not add anything useful to my experience. I'm not waiting for every typed character to appear on my screen before I type next one.
If I'm expecting data from the server (say I'm searching through shell history), it might even make my experience worse because of flickering between local and then remote state.
I'm regularly working with 100-200 ms latency servers where latency is noticeable. Improving my perceptual latency was never something I want.
Better approach is to use client/server software. Like vscode. Of course it's out of capabilities of terminal emulator and rather requires full rethinking of the entire shell architecture. But at least it would be meaningful improvement. Like I'm typing first character in shell history search, server transfers all 100 matched lines to the client and now my client can further filter this list without any server hops.
It doesn't really fully replace mosh since it doesn't do any of the perceptual lag reduction, but I do use SSH over Wireguard and it is fairly stable. It can indeed survive roaming.
I use ZNC over wireguard and it works fairly well! I can travel in the car with my phone connected to IRC, roam from my home WiFi to the cellular network and back without disconnecting from ZNC! It even works pretty well when cellular service is spotty.
This isn't quite what you asked but I think it's relevant!
Try looking at options ServerAliveInterval and TCPKeepAlive, they should prevent SSH from detecting when there's temporary transport issues and persist when wireguard roams between IPs.
I gave mosh a shot on a whim, and it's just magic. The only time I have to re-run it is when the client is powered down. Otherwise, I've had a rock solid remote terminal for months, surviving VPN and network finickyness.
One issue that Mosh doesn't handle well bad lag/high latency. It's great for automatically reconnecting if you have a flaky connection, but it actually makes a slow connection into a worse experience because the local terminal isn't matching the remote session.
I wish I could use mosh, but unfortunately it can't use ProxyJump to tunnel through a bastion host which is how I always use SSH, which makes sense as it's use Proxy commands itself.
I have very mixed feelings about bastion hosts in general. It usually feels like moving the problem around, eventually you end up adding even more complexity just to do your job. Wireguard (or Zerotier/Tailscale, depending on your threat model) might end up being simpler. Complex is the enemy of secure - if it's hard to use, then it's easy to make a mistake.
I tend to agree with you, but we don't always have control over the networks we access, and fitting in with other org's policies is often required. ProxyJump is very handy for that.
There is a way [1] but the first hop will be ssh/tcp which may defeat the point of doing this if the first hop has high loss or the IP changes constantly. ncat from nmap can also be used within SSH vs their suggested fifo method. Both have some caveats. Here [2] is some further discussion.
Agreed. I think the only use cases would be that your fist hop is stable but you want to proxy to another region over a lossy link but I can't imagine that would be very common. Maybe when using a satellite link or a inter-continental connection has a single over-saturated link but that would be a very fringe case. Perhaps when Starlink gets over-subscribed that could be a case.
I forgot why but I installed ET (painstakingly by creating a Gentoo ebuild and RC scripts…) then realized I don’t want it. I wish I could remember why.
Used to like mosh but have since switched over to Eternal Terminal[1] and will never go back. The scrollback alone is awesome.
[1]: https://eternalterminal.dev/
I run screen inside my mosh sessions which provides scrollback, among other features. (Tmux is another option.) ET is also a great solution, but I happen to need screen or something like it for its other features anyway.
I've been using mosh since it was announced, but I periodically try Eternal Terminal.
However I havent ever managed to get ET running on Amazon Linux which all my AWS EC2 instances run. I've tried compiling, docker, app images and nothing has worked.
Just checked this out, and the scrollback feature looks great.
I'd like to try this out, but am I understanding correctly that this needs to be installed on the server side as well as the client? That seems like a fairly large limitation compared to mosh.
Mosh also needs to be installed on the server.
Ah, you're right, I misunderstood. Thanks!
Did they disable the telemetry? Last time I checked, you needed to compile it yourself to get rid of that.
Not a fan of telemetry in secure shell tools.
The first message I get when starting it up:
Mosh has been sort of semi-neglected in a while - they did merge PRs, but did not release stable in years - but very recently, they started releasing new versions again.
I love to use it when I can.
I have been using mosh on and off, and actually appreciated that the software was so robust that it just works without frequent updates, and as far as I understand, most security is in ssh, so there would not really be a need for a new version.
Your understanding is sort of mistaken. Mosh does an initial handshake over ssh, but after that it’s a custom UDP protocol. There have been and could be more security bugs in that UDP protocol.
The only mosh CVE [1] was in the terminal emulator (a DoS that could only be triggered by a local user), not in the protocol. There have been no vulnerabilities in mosh's UDP protocol.
[1] https://nvd.nist.gov/vuln/detail/CVE-2012-2385
yet.
I wonder if anyone's thrown a fuzzer at it.
Yes, mosh has fuzz tests in oss-fuzz [1].
[1] https://github.com/google/oss-fuzz/tree/master/projects/mosh
Ahh, thanks for clarifying! Yes, in that case it is very appreciated to see recent updates.
Truecolor support was added in the recent 1.4 release which is quite nice.
Somewhat related...
On Wireguard's homepage[1] it says "[Wireguard] is even capable of roaming between IP addresses, just like Mosh. There is no need to manage connections, be concerned about state, manage daemons, or worry about what's under the hood."
Has anyone been using Wireguard as a successful Mosh replacement? Would love to hear your experience.
[1] https://www.wireguard.com/
Edit: added link
It kinda works, but mosh reconnects on failure regardless of how long time has passed. Since WG is a layer below, the TCP connection eventually gets reset. It is a huge difference. Also mosh works around lag by local echo, no such thing in WG.
Why local echo is useful?
If I'm expecting echo, it does not add anything useful to my experience. I'm not waiting for every typed character to appear on my screen before I type next one.
If I'm expecting data from the server (say I'm searching through shell history), it might even make my experience worse because of flickering between local and then remote state.
I'm regularly working with 100-200 ms latency servers where latency is noticeable. Improving my perceptual latency was never something I want.
Better approach is to use client/server software. Like vscode. Of course it's out of capabilities of terminal emulator and rather requires full rethinking of the entire shell architecture. But at least it would be meaningful improvement. Like I'm typing first character in shell history search, server transfers all 100 matched lines to the client and now my client can further filter this list without any server hops.
It doesn't really fully replace mosh since it doesn't do any of the perceptual lag reduction, but I do use SSH over Wireguard and it is fairly stable. It can indeed survive roaming.
I use ZNC over wireguard and it works fairly well! I can travel in the car with my phone connected to IRC, roam from my home WiFi to the cellular network and back without disconnecting from ZNC! It even works pretty well when cellular service is spotty.
This isn't quite what you asked but I think it's relevant!
Try looking at options ServerAliveInterval and TCPKeepAlive, they should prevent SSH from detecting when there's temporary transport issues and persist when wireguard roams between IPs.
I gave mosh a shot on a whim, and it's just magic. The only time I have to re-run it is when the client is powered down. Otherwise, I've had a rock solid remote terminal for months, surviving VPN and network finickyness.
Related:
Mosh: The Mobile Shell - https://news.ycombinator.com/item?id=28150287 - Aug 2021 (153 comments)
Mosh: the mobile shell - https://news.ycombinator.com/item?id=12429203 - Sept 2016 (49 comments)
Mosh: the mobile shell - https://news.ycombinator.com/item?id=11572146 - April 2016 (147 comments)
Mosh – a robust, responsive replacement for SSH - https://news.ycombinator.com/item?id=8928506 - Jan 2015 (45 comments)
Mosh: A replacement for SSH - https://news.ycombinator.com/item?id=8252093 - Sept 2014 (122 comments)
Mosh (mobile shell) - https://news.ycombinator.com/item?id=6321474 - Sept 2013 (6 comments)
Mosh: the mobile shell - https://news.ycombinator.com/item?id=5016745 - Jan 2013 (89 comments)
Mosh: the mobile shell - https://news.ycombinator.com/item?id=4588239 - Sept 2012 (1 comment)
Mosh: SSH for 2012 - https://news.ycombinator.com/item?id=3819382 - April 2012 (193 comments)
One issue that Mosh doesn't handle well bad lag/high latency. It's great for automatically reconnecting if you have a flaky connection, but it actually makes a slow connection into a worse experience because the local terminal isn't matching the remote session.
Semi-related: I built Muxile so you can connect to your running Tmux instance through your mobile. https://github.com/bjesus/muxile
I wish I could use mosh, but unfortunately it can't use ProxyJump to tunnel through a bastion host which is how I always use SSH, which makes sense as it's use Proxy commands itself.
I have very mixed feelings about bastion hosts in general. It usually feels like moving the problem around, eventually you end up adding even more complexity just to do your job. Wireguard (or Zerotier/Tailscale, depending on your threat model) might end up being simpler. Complex is the enemy of secure - if it's hard to use, then it's easy to make a mistake.
I tend to agree with you, but we don't always have control over the networks we access, and fitting in with other org's policies is often required. ProxyJump is very handy for that.
Mosh runs directly over UDP, and that's sort of essential to how it works, which is why it can't run through a (SSH + TCP) proxy host.
There is a way [1] but the first hop will be ssh/tcp which may defeat the point of doing this if the first hop has high loss or the IP changes constantly. ncat from nmap can also be used within SSH vs their suggested fifo method. Both have some caveats. Here [2] is some further discussion.
[1] - https://superuser.com/questions/53103/udp-traffic-through-ss...
[2] - https://teddit.sethforprivacy.com/r/linuxadmin/comments/xeqp...
Yeah, I think that defeats much of the purpose of using mosh. :)
Agreed. I think the only use cases would be that your fist hop is stable but you want to proxy to another region over a lossy link but I can't imagine that would be very common. Maybe when using a satellite link or a inter-continental connection has a single over-saturated link but that would be a very fringe case. Perhaps when Starlink gets over-subscribed that could be a case.
Love it but ran into a problem running multiple mosh connections in different windows, which i lounge for keeping things separate.
Any suggestions?
I wish tmux -CC worked as well as mosh did. I really prefer tmux as native GUI widgets.
I’ve had great success using “tmux -CC” / iTerm’s native integration with EternalTerminal.
I forgot why but I installed ET (painstakingly by creating a Gentoo ebuild and RC scripts…) then realized I don’t want it. I wish I could remember why.
Do you then go et server tmux -CC?