Keep coding agents running on a remote machine when SSH drops

Running agents on a remote Linux machine has real advantages: more cores, a laptop you can close, an environment that matches production. The usual first attempt is to SSH in, start the agent and walk away. Then the Wi-Fi blinks or the lid closes, and the agent is gone, halfway through an edit.

Three things can end a remote agent: the hang-up when the connection drops, the end of your login session, and the end of your last session. Each has its own fix, and a fourth setting tells you sooner when a connection has died. This guide takes them in the order you will hit them.

What kills an agent when the connection drops

When the SSH connection ends, the login shell on the server receives SIGHUP, the hang-up signal. The bash manual describes what follows: the shell exits by default, and before exiting an interactive shell resends the SIGHUP to all its jobs, running or stopped. The agent is one of those jobs, so it exits too. disown or nohup can stop the signal, but an agent is an interactive terminal program: even if it survives, there is no terminal left to answer its next question in.

What you want is a terminal that lives on the server and outlives the connection. That is what tmux is.

1. Run agents inside tmux

Start the tmux server once on the remote machine. On a Linux machine that uses systemd, start it through systemd-run, for the reason in step 2; elsewhere, plain tmux new -A -s agents does the same job.

# Create the session called "agents", outside your login session
systemd-run --scope --user tmux new -A -s agents

# Inside it: one window per agent, each in its own worktree
tmux new-window -n auth -c ~/src/app-auth
tmux new-window -n billing -c ~/src/app-billing

-A makes new-session attach if the session already exists, so the same command is safe to run twice. Detach with Ctrl-b d, or just lose the connection: the tmux server keeps the windows, and the agents in them keep running. To reconnect from your laptop in one step:

ssh -t devbox tmux attach -t agents

That attaches to the running server instead of starting a new one inside the SSH session, which would undo step 2. If attach reports no server, start it again with the systemd-run line.

If two agents work on the same repository, give each its own git worktree, for the reasons in our guide to worktrees.

2. Why tmux starts outside your login session

On Linux machines that use systemd, logging out can end every process in the session, tmux included. The setting is KillUserProcesses in logind.conf. The systemd documentation lists its default as "yes" and warns that it "will break tools like screen(1) and tmux(1), unless they are moved out of the session scope". Distributions can change that default, so check yours:

systemd-analyze cat-config systemd/logind.conf | grep KillUserProcesses

A commented-out line means the compiled-in default applies. systemd-run --scope --user starts tmux under your systemd user manager instead of inside the login session, as the systemd-run manual shows for screen. Only the command that starts the tmux server matters: the server stays where it was started. If you already have a server that was started with plain tmux, it is still inside a login session. Finish what is running in it, run tmux kill-server, and start it again with systemd-run.

3. Turn on lingering

Moving tmux under the user manager is not the end of it. The systemd-run manual explains that the user manager starts at your first login and stops after your last session ends, taking everything under it along. Lingering keeps it running:

loginctl enable-linger

With lingering on, a user manager is started for you at boot and kept around after you log out. systemd's default policy lets users turn lingering on for themselves; if your machine's policy refuses, an administrator can run sudo loginctl enable-linger <user>. Steps 2 and 3 together are what let an agent work through the night with nobody logged in.

4. Notice dead connections quickly

Without keepalives, a connection that dies quietly (a laptop that went to sleep, a network that changed) can leave your terminal frozen until TCP gives up. OpenSSH can check the connection itself. In ~/.ssh/config on your laptop:

Host devbox
  HostName devbox.example.internal
  ServerAliveInterval 15
  ServerAliveCountMax 3

ServerAliveInterval sends a message through the encrypted channel after 15 seconds of silence, and ServerAliveCountMax is how many may go unanswered. The ssh_config manual works the example: with these values an unresponsive server is dropped after about 45 seconds. Your agents are unaffected, because they live in tmux; you just get your prompt back and reconnect.

See who is waiting from the status line

Hooks run on the machine the agent runs on, so a desktop notification on the server never reaches your laptop. The state files from our guide to agent hooks work well here: put a count in tmux's status line, which you see from every window.

# ~/.tmux.conf on the server
set -g status-interval 5
set -g status-right '#(grep -l needs-input "${XDG_STATE_HOME:-$HOME/.local/state}"/agent-state/* 2>/dev/null | wc -l | tr -d " ") need you'

tmux runs the #() command with /bin/sh and shows the last line of its output. By default the status line refreshes every 15 seconds; 5 is a reasonable compromise.

Two smaller things

• Sign in on the server. Each agent CLI keeps its own sign-in on the machine it runs on. Log in to each one once on the remote host.
• Check what is still running. tmux ls lists sessions, and tmux list-windows -a lists every window in all of them. After a reboot nothing is running until you start tmux again; lingering keeps processes alive across logouts, not across restarts.

The whole setup

1. Start the tmux server once: systemd-run --scope --user tmux new -A -s agents.
2. Enable lingering once: loginctl enable-linger.
3. Add ServerAliveInterval and ServerAliveCountMax to your SSH config.
4. Reconnect with ssh -t devbox tmux attach -t agents.

Tenslo Code handles this for projects on SSH hosts: agents and terminals there keep running when the connection drops, and the app reconnects to the machine on its own.