Can SSH pipeline exit status hide a failed command?

A remote command can fail, a formatter can print convincing output, and an agent can still announce success. That is not an SSH mystery. It is ordinary shell semantics crossing a network boundary without enough evidence attached.

The fix is not merely adding set -o pipefail to every script. pipefail changes one aggregate result. An agent that runs consequential SSH work needs the status of each pipeline stage, a defined rule for expected nonzero exits, and a final remote exit code that cannot be mistaken for success. Capture the vector immediately, name it, and make the wrapper decide what success means.

A green final command can hide a red first command

By default, a shell pipeline reports the exit status of its final command. That makes this line dangerous in deployment, migration, backup, and repair work:

build_manifest | sign_manifest | tee /var/tmp/manifest.json

Suppose build_manifest fails because it cannot read a required file. sign_manifest might receive no usable input and fail too, or it might produce an empty result. tee can still create a file, write zero bytes, and exit with status zero. The shell reports zero for the whole pipeline. A caller that checks only $? sees success.

The GNU Bash Reference Manual states this plainly: a pipeline uses the exit status of the last command unless pipefail is enabled. Bash waits for all commands in a synchronous pipeline, but waiting is not the same as preserving their outcomes.

A human at an interactive terminal sometimes notices the missing data or error message. An agent often has a narrower view. It may receive a truncated transcript, a formatted summary, or only the command's final result. If the script says zero, the agent has grounds to say the action succeeded even when the action did not do the thing requested.

The distinction that teams blur is simple:

• A pipeline's exit status is one decision value.
• The statuses of its commands are the evidence behind that value.

You need both. The decision value controls whether the remote command returns success. The evidence tells a reviewer, a log, or a supervising agent where it went wrong.

This matters most when the first stage changes the world. Consider a remote export that reads production data, compresses it, encrypts it, and uploads it. The upload client may exit zero after uploading an empty stream. The transcript can contain reassuring words such as "completed" because a later program completed its narrow job. That result must not become a false statement that the export succeeded.

SSH returns what the remote shell chooses to return

OpenSSH does not inspect the commands inside a remote shell pipeline. It returns the status of the remote command, or 255 when SSH itself encounters an error.

That behavior is correct and useful. SSH cannot know whether this remote text is a pipeline, a shell function, a script, or an application that uses exit codes in its own way:

ssh deploy@host 'generate | transform | tee result.txt'

The remote login shell parses that command. If its pipeline semantics report the final tee status, then SSH returns that status to the local machine. The local caller has no way to reconstruct the earlier results after the remote shell has discarded them.

Putting set -o pipefail in the local shell does not repair a pipeline that runs remotely. This command changes only the status rules of the local pipeline:

set -o pipefail
ssh deploy@host 'generate | transform | tee result.txt'

The remote shell still owns generate | transform | tee result.txt. It needs its own explicit shell and its own failure handling.

There is a second trap. This local command creates another pipeline after SSH returns:

ssh deploy@host 'remote command' 2>&1 | tee session.log

Now two different pipelines exist:

1. The remote shell may have a pipeline inside remote command.
2. The local shell has ssh | tee session.log.

A successful local tee can hide an SSH transport failure or a nonzero exit from the remote wrapper. You must inspect the remote pipeline on the host and the local pipeline around SSH. Treating the line as one opaque command is how false green results survive review.

Pipefail detects a failure but does not explain it

set -o pipefail changes Bash's aggregate result. With it enabled, Bash returns the status of the rightmost command that exited nonzero, or zero if every command succeeded.

For many scripts, that is a real improvement:

set -o pipefail
produce_data | validate_data | publish_data
printf 'pipeline status: %s\n' "$?"

If produce_data exits 17 and the later commands exit zero, the pipeline returns 17. If validate_data exits 4 and publish_data exits zero, the pipeline returns 4. Your calling process receives a failure rather than a lie.

But pipefail loses detail when more than one stage fails. Suppose the statuses are 17 4 0. The pipeline result is 4, because 4 came from the rightmost failed command. That tells you failure occurred, but it does not prove whether the validator caused the producer failure, reacted to it, or failed independently.

That is why pipefail is a guardrail, not a report format. Use it when you want a pipeline to fail as a unit. Use PIPESTATUS when you need to answer these questions later:

• Which stage returned a nonzero code?
• Did a later stage run and succeed after an earlier stage failed?
• Did the process receive a signal rather than return its own error?
• Is a nonzero code expected for this particular command?

Do not paper over the gap with || true:

produce_data | validate_data | publish_data || true

That pattern is popular because it keeps a script moving. It also erases the only signal the caller had. If a stage may legitimately return nonzero, encode the allowed status for that stage after you have captured the actual vector. Do not waive failure for the whole pipeline.

PIPESTATUS disappears if you wait even one command

Bash exposes each stage's exit code in the PIPESTATUS array. The array is fragile by design: it describes the most recently executed foreground pipeline, and the next command can replace it.

This looks reasonable but is wrong:

source_data | normalize | upload
pipeline_rc=$?
printf 'pipeline result: %s\n' "$pipeline_rc"
statuses=("${PIPESTATUS[@]}")

By the time Bash reaches the final assignment, the pipeline_rc=$? assignment and printf have run. PIPESTATUS no longer describes source_data | normalize | upload.

Copy the array first, before doing anything else:

source_data | normalize | upload
statuses=("${PIPESTATUS[@]}")

Then inspect it without relying on the pipeline's aggregate code:

printf 'source_data=%s normalize=%s upload=%s\n' \
  "${statuses[0]}" "${statuses[1]}" "${statuses[2]}"

This is also why a casual set -e can make diagnosis worse. With pipefail active, a failed pipeline may cause Bash to exit before the next line copies PIPESTATUS. Shell error handling has several contextual exceptions, and scripts that depend on set -e alone often produce less evidence precisely when a command fails.

For a pipeline whose statuses matter, turn off errexit for the few lines needed to run and snapshot it. Then make an explicit decision. That is more code than a magic shell option, but it is code you can read during an incident.

Run the remote program under the shell you require

PIPESTATUS is a Bash array. It is not portable POSIX sh syntax, and pipefail is not required by POSIX shell. A remote command invoked through SSH may run under a login shell you did not choose. On one host it may be Bash; on another it may be dash, zsh, or a restricted shell.

Do not send Bash syntax to an unspecified remote shell and hope the machine happens to agree with you. Start Bash explicitly:

ssh deploy@host 'bash -s' <<'REMOTE_SCRIPT'
printf 'alpha\n' | grep 'beta' | tee /var/tmp/example.out
statuses=("${PIPESTATUS[@]}")
printf 'stages=%s,%s,%s\n' \
  "${statuses[0]}" "${statuses[1]}" "${statuses[2]}" >&2
REMOTE_SCRIPT

The quoted heredoc delimiter matters. <<'REMOTE_SCRIPT' prevents the local shell from expanding variables, command substitutions, and backslashes before it sends the script. The remote Bash process receives the text you wrote.

On macOS, the system Bash is old but it supports indexed arrays, PIPESTATUS, and set -o pipefail. That does not mean /bin/sh is Bash. A script with #!/bin/bash helps only when you execute that file directly. If you pass a one-line command to ssh host '...', the remote login shell still parses it unless you explicitly start Bash.

For a maintained automation path, put the remote wrapper in a versioned script and invoke its absolute path. For short-lived agent work, bash -s with a quoted heredoc is usually easier to audit because the full remote program appears in the local action request.

A wrapper should name stages and return an honest result

A useful remote wrapper does four jobs. It runs the pipeline, copies the status vector immediately, emits a machine-readable record, and exits nonzero when a required stage failed.

This example uses a three-stage data transfer. Replace the commands, but keep the control flow. It deliberately does not rely on set -e to decide what happens after the pipeline.

#!/usr/bin/env bash
set -uo pipefail

run_export() {
  local -a status
  local stage
  local -a names=("collect" "compress" "send")

  set +e
  collect_records | gzip -c | send_archive --destination daily
  status=("${PIPESTATUS[@]}")
  set -e

  if ((${#status[@]} != ${#names[@]})); then
    printf 'agent_pipeline_error pipeline=export reason=status_count expected=%s got=%s\n' \
      "${#names[@]}" "${#status[@]}" >&2
    return 70
  fi

  for stage in "${!names[@]}"; do
    printf 'agent_pipeline_status pipeline=export stage=%s code=%s\n' \
      "${names[$stage]}" "${status[$stage]}" >&2
  done

  for stage in "${!status[@]}"; do
    if (( status[stage] != 0 )); then
      printf 'agent_pipeline_result pipeline=export outcome=failed\n' >&2
      return "${status[$stage]}"
    fi
  done

  printf 'agent_pipeline_result pipeline=export outcome=ok\n' >&2
  return 0
}

run_export

A failed collection with a successful compressor and sender produces output in this shape:

agent_pipeline_status pipeline=export stage=collect code=23
agent_pipeline_status pipeline=export stage=compress code=0
agent_pipeline_status pipeline=export stage=send code=0
agent_pipeline_result pipeline=export outcome=failed

The wrapper exits 23. SSH returns 23 to the local process. The agent can report that the export failed at collect, even if send_archive printed a completion message for an empty stream.

The exact returned code is less important than the discipline behind it. In this wrapper, the first nonzero status in pipeline order wins. Bash pipefail instead selects the rightmost nonzero status. Either policy can work if you state it and test it. I prefer the first failed stage for operations work because it usually points closer to the initiating fault. Keep the entire status vector in the action record so nobody must infer what happened from a single number.

The stage names are not decoration. 0=23,1=0,2=0 forces a person to reopen the script. collect=23,compress=0,send=0 lets a supervisor route the failure, add context, or decide whether a retry is safe.

Local logging can create a second false success

Operators want a local transcript. Agents need one too. The naive way to get it is this:

ssh deploy@host 'bash -s' < remote-export.sh 2>&1 | tee ssh-export.log

If SSH returns 23 but local tee writes the transcript and returns zero, the local pipeline returns zero by default. You fixed the remote lie and introduced a local one.

Capture the local statuses as well:

set +e
ssh deploy@host 'bash -s' < remote-export.sh 2>&1 | tee ssh-export.log
local_status=("${PIPESTATUS[@]}")
set -e

ssh_rc=${local_status[0]}
tee_rc=${local_status[1]}
printf 'ssh=%s tee=%s\n' "$ssh_rc" "$tee_rc" >&2

if (( ssh_rc != 0 )); then
  exit "$ssh_rc"
fi
if (( tee_rc != 0 )); then
  exit "$tee_rc"
fi

Do not enable local pipefail and stop there. It gives a nonzero aggregate result if either ssh or tee failed, which is better than default behavior. It cannot tell the agent whether the remote action failed, the network connection failed, or local logging failed. Those cases lead to different decisions.

An SSH status of 255 needs special treatment. OpenSSH reserves it for an error in the SSH client path, rather than a remote command result. A wrapper should report it as a transport or SSH execution failure, not claim that a named remote pipeline stage returned 255.

There is another practical reason to keep local and remote results separate. A transcript may contain several remote pipeline records, warnings from the login shell, and an SSH diagnostic. If an agent parses free-form text for the last number it sees, it will eventually pick the wrong number. Use recognizable records, then bind the final action result to the actual process exit status.

SIGPIPE needs a written exception, not a blanket pardon

pipefail exposes a failure that many scripts previously ignored: SIGPIPE. In Bash, a process terminated by signal number N gets a status of 128 + N; SIGPIPE commonly appears as 141.

A classic intentional case is:

generate_many_lines | head -n 10

head reads ten lines and exits successfully. The generator may continue writing, receive SIGPIPE because no reader remains, and exit 141. With pipefail, the pipeline can look failed even though the requested ten-line sample was produced.

That does not make 141 harmless in every pipeline. A network client, compressor, or data producer can receive SIGPIPE because an unexpected downstream consumer crashed or rejected input. If you mark all 141 statuses as success, you conceal a broken transfer.

The right rule is narrow: allow a signal-derived status only for a stage whose early termination is part of the command's intended contract. Put that exception beside the stage, not in a global shell setting.

For example, a wrapper around a deliberate preview can accept generate_many_lines=141 only when head=0:

if (( status[0] == 141 && status[1] == 0 )); then
  printf 'agent_pipeline_result pipeline=preview outcome=ok reason=expected_sigpipe\n' >&2
  return 0
fi

Every other nonzero result remains a failure. This small amount of specificity prevents a common overcorrection: people enable pipefail, see a noisy 141 once, then disable it across the entire automation estate.

An agent needs evidence separate from command output

The agent should not determine success by reading prose. Commands print success words before they fail, tools mix warnings with results, and a remote script may emit a final line after one stage has already gone wrong.

Define an action contract with two layers:

1. The process exit code decides whether the requested action succeeded.
2. Structured status records explain every pipeline stage that mattered.

Keep ordinary command output available for debugging, but do not ask the agent to infer control flow from it. In the wrapper above, stderr carries records that begin with agent_pipeline_status and agent_pipeline_result. A calling program can preserve that stream, parse only those exact records, and still show the rest to a human.

Do not trust a marker just because it appears in untrusted command output. If a pipeline stage handles data supplied by another user or system, that data can contain a line resembling your status record. The safer pattern is to have the wrapper capture stage output and emit the records itself after the pipeline finishes. For higher-risk work, use a dedicated result file with restrictive permissions, then have the wrapper read and validate it before emitting one final record.

The agent's report should include the remote exit code, the local SSH exit code, and the named remote stage statuses when available. It should also distinguish these outcomes:

• the remote action ran and a named stage failed;
• the remote wrapper could not produce a complete status record;
• SSH could not establish or maintain the action channel;
• local transcript capture failed after the remote action completed.

Those are operationally different facts. A retry after a network loss may duplicate a completed remote mutation. A retry after a failed validation stage may be safe. A retry after a failed local tee may be pointless because the remote work already happened.

Sallyport can keep the SSH credential out of the agent while it executes the SSH action, but the remote command still needs this honest exit and evidence contract.

Test the failure paths before an agent reaches them

A shell wrapper earns trust only after it fails in controlled ways. Testing the happy path proves the least interesting branch.

Create disposable commands that return the statuses you want to observe:

fail_23() { printf 'collector failed\n' >&2; return 23; }
pass_through() { cat; }
succeed() { cat >/dev/null; return 0; }

set +e
fail_23 | pass_through | succeed
status=("${PIPESTATUS[@]}")
set -e
printf 'observed=%s,%s,%s\n' "${status[0]}" "${status[1]}" "${status[2]}"

The expected result is 23,0,0. Then run the same pattern through the exact SSH invocation your agent uses. Do not stop at a local shell test, because the remote shell selection, heredoc quoting, local transcript pipeline, and wrapper exit behavior all sit outside that first check.

Test at least these cases:

• every stage succeeds and the wrapper exits zero;
• an early stage fails while later stages return zero;
• a middle stage fails after consuming some input;
• SSH cannot connect or authenticate;
• local tee cannot write its transcript;
• a deliberate head pipeline triggers the expected SIGPIPE rule.

Record the expected exit code and the expected stage records for each. If a test says success after an earlier stage returns nonzero, the wrapper has not done its job.

The tempting shortcut is to make the agent inspect a transcript after every action and judge whether the output "looks right." That fails under load, fails when tools change their wording, and fails when output is truncated. Exit codes are the control channel. Stage records are the evidence channel. Keep them separate, preserve both across SSH, and do not let a final tee decide whether a remote action happened.