SSH startup files: why remote commands lie to agents

An SSH command is not automatically the program you typed. Before the remote machine runs git, python, or a deployment script, the SSH daemon selects an account, invokes that account's shell, and gives that shell a command string. Startup files and server rules can change the environment, replace the command, or make one invocation interactive while another stays noninteractive.

That matters more when an agent reports the result. A human who sees an unexpected banner, alias, or colored prompt may stop and investigate. An agent can read exit code 0 and a familiar-looking line of output as proof that the intended program ran. I have seen that assumption turn a harmless status check into a call to a wrapper script, and a deployment check into a result from the wrong executable.

The fix is not to delete every profile file. People need usable interactive shells. The fix is to identify the execution path, separate human convenience from automation, and make the remote command contract explicit.

A remote command first passes through the account shell

OpenSSH does not normally execute the words after ssh host as a direct execve of the target binary. The sshd manual says that, after authentication, sshd runs the requested command through the user's shell with the shell's -c option. The login shell path comes from the account database, not from the local machine that opened the SSH connection.

That detail explains a lot of confusing reports. Suppose an agent sends:

ssh deploy@buildbox git -C /srv/app rev-parse HEAD

The remote account might use Bash, zsh, fish, a restricted shell, or a site wrapper. The selected shell receives a command equivalent to git -C /srv/app rev-parse HEAD. It can initialize itself first. It can also receive a replacement command from sshd before it gets that far.

This is separate from the shell that the agent used locally. An agent may run from a clean process on a Mac while the remote deploy account has ten years of personal shell customizations. A clean local prompt does not make the far end clean.

Do not confuse a shell's command parsing with the SSH transport either. SSH encrypts the connection and authenticates the account. It does not promise that python means the binary you expected, that PATH stayed unchanged, or that a profile did not print text into standard output.

The first practical question is therefore not "Did SSH connect?" It is "Which program interpreted the remote command, under which account, and with which initialization rules?"

Shell mode decides which files get a vote

Shell startup behavior depends on shell family and invocation mode. The labels people casually use, "an SSH shell" or "a Bash shell," are not precise enough to predict behavior.

For Bash, the GNU Bash Reference Manual distinguishes login, interactive, and noninteractive invocation. A login Bash shell reads /etc/profile, then the first readable personal file among ~/.bash_profile, ~/.bash_login, and ~/.profile. An interactive shell that is not a login shell reads ~/.bashrc.

A normal remote command usually runs under a noninteractive shell. That does not mean it loads nothing. Bash reads the file named by BASH_ENV, if that environment variable is set, before executing a noninteractive command. The Bash manual also documents special handling when Bash detects that sshd started it with its standard input connected to a network connection: it may read ~/.bashrc. Do not build automation on a folk rule that says .bashrc only affects interactive use.

zsh has a different map. /etc/zshenv and ~/.zshenv apply to every zsh invocation, which makes a noisy or stateful .zshenv particularly dangerous. zsh reads .zprofile for login shells and .zshrc for interactive shells. A user who puts PATH edits and status output in .zshenv has changed remote commands, scripts, and often graphical application launches as well.

POSIX sh does not give you a universal escape hatch. Implementations differ. The POSIX shell specification describes the ENV file for interactive shells, but a system's /bin/sh may be dash, Bash in compatibility mode, or another implementation with its own details. Test the actual host instead of assuming that a filename means the same thing everywhere.

The distinction has a hard operational consequence. A command that succeeds in an interactive troubleshooting session can fail for an agent because the interactive session reads a profile that the command session skips. The reverse also happens: an automation hook can affect command sessions while a human's terminal misses it.

Keep interactive settings in files that only interactive shells read. Prompts, completion setup, terminal titles, color defaults, and greetings belong there. Put automation environment requirements in a small, documented file that a controlled launcher sources deliberately. Do not let an agent discover its runtime by inheriting a developer's dotfiles.

PATH changes alter identity, not just convenience

PATH is often treated as a convenience setting. In unattended execution, it selects program identity. If a profile prepends /opt/team/bin, then curl, git, ssh, or python may refer to a wrapper instead of the system program.

That wrapper may be intentional. Teams use wrappers to choose cloud credentials, enforce repository checks, add telemetry, or select language runtimes. The mistake is allowing an agent to infer that an unqualified command means the vendor executable. It only means "the first matching executable in the PATH that this shell currently has."

Aliases and functions create similar ambiguity, though their behavior depends on shell mode. Bash normally does not expand aliases in a noninteractive shell unless expand_aliases is enabled. That makes aliases less common in standard SSH command runs, not impossible. A sourced file can enable expansion. Functions do not need alias expansion at all. A profile can define a function named git, export environment state, and arrange for every later command in that shell to call the function.

Start by asking the shell, but treat its answer as evidence from the environment you are inspecting, not as a universal proof:

ssh -T deploy@buildbox '
printf "shell argv0: %s\n" "$0"
printf "shell flags: %s\n" "$-"
printf "PATH: %s\n" "$PATH"
command -V git
command -V python
command -V curl
'

A typical result might look like this:

shell argv0: -bash
shell flags: hBc
PATH: /opt/team/bin:/usr/local/bin:/usr/bin:/bin
git is /opt/team/bin/git
python is /usr/local/bin/python
curl is /usr/bin/curl

The command -V builtin can report an alias, function, builtin, or pathname. On shells that support it, type -a git can display more than one matching path. In Bash, declare -f git prints a function body if git names a function. Those checks expose a common failure: a profile defines a git() function that runs a real Git command and then quietly sends a status event. The output still looks like Git output. The side effect is elsewhere.

Command hashing adds another wrinkle. Some shells cache where they found a command. If a profile changes PATH after the shell has resolved a name, a shell may continue using the cached location until a hash -r or its equivalent clears the cache. This mostly appears in long-lived interactive shells, but agents that keep a shell process alive can hit it too.

Use absolute paths for commands whose identity affects a security decision or deployment result. Do not write /usr/bin/git merely because it exists on your laptop. Verify the expected path on the target operating system, record it, and test it under the account that will run the work. If the deployment deliberately uses a version manager or wrapper, name that wrapper explicitly and make its contract part of the job.

A pseudo-terminal changes more than formatting

ssh host command normally does not allocate a pseudo-terminal. ssh -t host command forces one. That single option can switch shell startup branches, affect buffering, cause programs to emit color codes, and prompt for input that a non-TTY run never requests.

Many shell configuration files start with a test like this:

case $- in
  *i*) ;;
  *) return ;;
esac

That check exits from the rest of the file unless the shell identifies itself as interactive. A pseudo-terminal can make a shell interactive in circumstances where an ordinary command connection was not. The command now gets aliases, completion setup, prompt helpers, PATH edits, and terminal-specific exports that were absent in the agent run.

Programs also react directly to the terminal. A command may show a progress display, paginate output, choose a different diagnostic format, or read confirmation input. A machine parser that expects one JSON object can fail because a profile prints a greeting before the program starts, or because the program detects a TTY and emits control characters.

Test both modes when investigating a mismatch:

ssh -T deploy@buildbox 'printf "flags=%s tty=" "$-"; test -t 1 && echo yes || echo no'
ssh -tt deploy@buildbox 'printf "flags=%s tty=" "$-"; test -t 1 && echo yes || echo no'

The first command asks SSH not to allocate a terminal. The second forces one even when the local standard input is not a terminal. Compare the output, then compare command resolution and PATH in each mode. If results differ, do not patch the parser first. Find the startup branch that made them differ.

Automation should default to -T. Allocate a terminal only for a command that genuinely needs one, such as a controlled interactive recovery task. An agent that needs a terminal to run a routine status query is already executing in a less predictable environment.

Server rules can replace the requested command

Startup files are only one layer. The server can replace or constrain a command before the account shell sees it. If you inspect .bashrc and stop there, you can miss the rule that actually changed the result.

The sshd_config manual documents ForceCommand. An administrator can apply it globally or inside a Match block for a user, group, address, or other condition. With ForceCommand internal-sftp, for example, the server ignores ordinary shell commands and runs the internal SFTP service. With a custom wrapper, the wrapper receives the requested command context and decides what to do.

An SSH public-key entry can also carry a command="..." restriction in the account's authorized_keys file. This is common for backup accounts, repository access, file transfer, and narrowly scoped automation. It can be good security practice. It becomes a debugging problem when someone gives an agent a restricted credential and expects arbitrary command execution.

Environment controls belong in the same review. AcceptEnv tells sshd which client-supplied variables it accepts. SetEnv can set variables server-side. PermitUserEnvironment, when enabled, can allow an account's SSH environment file or authorized key options to set values. These settings can influence PATH, locale, language runtime behavior, and hooks such as BASH_ENV.

Use ssh -G on the client, but know its boundary:

ssh -G buildbox | grep -E '^(hostname|user|port|requesttty|remotecommand|sendenv|setenv) '

OpenSSH prints the client configuration after it has applied local Host blocks and defaults. The output can reveal an unexpected RemoteCommand, forced terminal setting, environment forwarding rule, or a different target host. It cannot reveal server-side ForceCommand, the remote account's login shell, or restrictions stored in authorized_keys.

Ask the server owner for the relevant sshd_config and account restrictions when the account is intended for automation. If you do not administer the host, ask for a documented execution interface instead of trying to reverse engineer personal shell behavior through trial runs. An account with an opaque forced wrapper is not a general SSH endpoint, even if it accepts authentication.

Inspect the execution path without trusting a single probe

A diagnostic command runs inside the very environment under suspicion. That does not make diagnosis impossible. It means you should collect several independent facts and label what each one proves.

First, inspect the local client configuration with ssh -G. Then determine the remote account's configured shell through the host's account database. On many Linux hosts, getent passwd deploy returns a colon-separated record whose final field is the shell path. On macOS, an administrator can inspect the account with Directory Service tools. Do this from a trusted administrative channel where possible, rather than through the account's potentially customized shell.

Next, retrieve and review the startup files for the actual shell: global files, personal files, and any files they source. Search for these kinds of behavior:

• PATH= assignments, version manager initialization, and hash commands
• alias, function definitions, and shell options such as expand_aliases
• output commands such as echo, printf, and terminal control helpers
• BASH_ENV, ENV, locale variables, and language-specific environment hooks
• conditional branches that test -t, $-, $SSH_TTY, or $TERM

Do not merely search for the word alias. A line such as . ~/.local/share/tool/init.sh may source the file that defines the function you care about. Follow the source chain until it ends. Profile files often grow into a pile of conditional includes, and that is where automation behavior becomes unreviewable.

Then perform a narrow fingerprint run with a harmless command. Record the shell flags, current directory, PATH, relevant environment names, and resolution of the exact tools the job will invoke. Avoid dumping every environment variable into logs. Tokens and cloud credentials frequently live there. A diagnostic transcript should prove execution conditions without becoming a new secret store.

Finally, test the exact production command in the exact mode the agent will use: same account, no terminal unless required, same input transport, and same working directory. A result obtained through an administrator's interactive login is not a substitute. It answers a different question.

Keep the evidence with the deployment or automation definition. The useful artifact is not a screenshot of a working terminal. It is a short document that names the account shell, startup files allowed to affect the run, required executable paths, expected environment, terminal policy, and server-side command restrictions.

Make the automation shell deliberately boring

A reliable remote action should enter a known shell with a short environment and then execute named binaries. This does not remove the initial account shell that sshd uses, but it limits how much inherited state reaches the program that does the work.

For a small fixed script, send the script over standard input and run a known shell with an empty environment. The remote command string stays fixed, which also avoids building a fragile tower of local and remote quote escaping.

ssh -T deploy@buildbox '/usr/bin/env -i PATH=/usr/bin:/bin /bin/sh -s' <<'REMOTE'
set -eu
PATH=/usr/bin:/bin
export PATH

/usr/bin/git -C /srv/app rev-parse HEAD
REMOTE

This has useful properties. -T avoids a terminal. /usr/bin/env -i clears inherited variables. /bin/sh -s reads the literal script from standard input, and the quoted heredoc prevents the local shell from expanding $PATH or other text before transmission. set -eu stops on an unset parameter or a failed simple command, subject to normal shell semantics.

It also has limits. /usr/bin/env, /bin/sh, and /usr/bin/git are examples, not paths to copy blindly across every host. Confirm the paths on each target class. Clearing the environment may remove variables a program legitimately needs, including a home directory, locale, proxy configuration, or runtime location. Add back only named variables that the program requires, and document why each one exists.

Do not use env -i as a way to hide a broken account configuration. If a startup file modifies the initial shell so aggressively that it prevents the fixed launcher from running, the account itself is not suitable for unattended work. Use a dedicated account with a controlled login shell and minimal startup files.

Avoid feeding arbitrary agent text into a remote sh -c string. Quoting mistakes can turn data into shell syntax before the intended script sees it. Pass data through a constrained channel, use a fixed remote script with validated arguments, or use a protocol designed for structured requests. Shell is powerful because it parses text as code. An agent should not get to blur that boundary by accident.

Separate interactive accounts from agent accounts

The cleanest design gives humans and automation different remote identities. A developer account can keep its prompt, completion, language managers, and personal helpers. An automation account should have a declared shell, a small home directory, and startup files that either do nothing for command sessions or run a narrowly reviewed setup.

This separation is not bureaucracy. It lets you answer mundane but necessary questions: Which Git binary deploys? Which directory is the starting directory? Does the account accept a terminal? Which environment names may affect a release? Who can change the wrapper that runs commands?

A dedicated account also makes command restrictions easier to reason about. You can use a forced wrapper for the few actions that account is meant to perform and reject everything else. That does not make the wrapper a policy language. Keep it simple enough to inspect: validate arguments, set the documented environment, invoke an absolute program path, and write an audit record.

Do not solve this by placing a large case statement with dozens of exceptions in .bashrc. That approach is popular because every user already has a profile file and no deployment change is needed. It fails because profile semantics vary by shell mode, source order becomes obscure, and a harmless interactive edit can alter unattended behavior. Put automation setup in the launcher or dedicated script that owns the automation contract.

Sallyport can keep SSH credentials out of an agent process and execute SSH actions through its bundled sp-ssh helper, but it cannot make an uncontrolled remote account deterministic. Treat the remote account review as part of the action design, then use the activity record to compare what the agent requested with what the remote side returned.

Trust results only after you pin down the contract

An agent command result is trustworthy when you can state what executed without relying on a person's current shell habits. That statement needs more than a hostname and an exit code. It should identify the SSH account, target host identity, terminal setting, server command restriction if any, selected shell, environment construction, script source, and executable path.

There is an awkward boundary here. You cannot prove from a remote command's own output that the remote shell did not alter the command before printing that output. If the account shell, authorized key restriction, or forced command is outside your control, you need administrative evidence or a different account. Repeating the same probe ten times only repeats the same assumption.

Start with the commands agents already run that can change code, infrastructure, or production data. For each one, run it with and without a terminal, inspect how its executable resolves, and replace inherited setup with an explicit launcher where that result matters. The first surprise is usually PATH. The second is usually a startup file someone forgot was still being sourced.

Once you have found one of those surprises, do not add another conditional to a profile and call it fixed. Move the behavior into a controlled account or fixed script, record the command contract, and make the next agent run boring enough that its output means what it says.