Why keep SSH stdout and stderr separate?

An SSH command result should preserve stdout, stderr, and termination as different facts. Flatten them into one string and an agent can no longer tell data from diagnosis, an operator cannot see why a command failed, and an audit viewer may present a sequence that never existed.

The fix is not a prettier delimiter. Keep the two byte streams separate, record a limited observation order when the transport exposes one, and represent exit status, exit signal, timeout, cancellation, and transport failure independently. That model takes a little more work at the capture boundary and removes a long list of parser bugs downstream.

SSH already distinguishes the two streams

SSH carries ordinary channel data and stderr through different protocol messages. RFC 4254 calls them SSH_MSG_CHANNEL_DATA and SSH_MSG_CHANNEL_EXTENDED_DATA; it assigns extended data type 1 to SSH_EXTENDED_DATA_STDERR. A client library that hands you separate readers is exposing a distinction the protocol deliberately retained.

That distinction carries meaning. Programs normally write machine-consumable results to stdout and diagnostics to stderr. A command can emit valid JSON on stdout while printing a warning on stderr and still return zero. Another can print partial output, describe a failure on stderr, and return nonzero. The bytes alone do not tell you which case occurred.

Merging at capture time throws away information that no later parser can recover. Prefixes such as [stderr] help a human, but they alter the content. Newline separators are worse: a chunk can end without a newline, binary data can contain any byte, and the added separator may turn two valid fragments into one invalid document.

Treat each stream as bytes until a consumer chooses a decoding policy. UTF-8 is common, but it is not guaranteed by SSH. Even nominally textual tools can emit invalid sequences because of locale mismatch, filenames containing arbitrary bytes, or a write cut off in the middle of a multibyte character. Store raw bytes or a lossless encoding, then provide decoded text as a view.

The protocol fact matters because it changes the burden of proof. If your result type has only output: string, the type is lying about what SSH delivered. Convenience rendering belongs after capture, where it can be replaced without rewriting the audit record.

Stream identity is not severity

Stderr means file descriptor 2, not failure. Treating every stderr byte as an error creates noisy agents that retry successful commands, discard usable stdout, or ask for approval after harmless warnings.

Many familiar programs use stderr for progress, verbose traces, prompts, and warnings. A compiler may reserve stdout for generated output while reporting its progress elsewhere. A command can also fail silently with a nonzero status. The relationship is useful evidence, not a Boolean rule.

Keep at least three concepts distinct:

• stdout and stderr identify where bytes arrived.
• exit_status or exit_signal describes how the remote program ended.
• transport_error describes whether the SSH operation itself completed.
• timed_out and cancelled describe local intervention.

This prevents a common parser mistake: stderr != empty becomes success = false. Success should normally mean that the command started, the channel completed, and the remote status was zero. Your application may apply a stricter rule for a particular command, but that belongs in the command adapter, not the generic SSH runner.

The reverse mistake is equally damaging. Some wrappers return only stdout on success and replace the entire result with an exception on failure. The exception may contain a clipped stderr suffix, while partial stdout disappears. An agent then sees less evidence precisely when it needs more.

Do not overload a single error field with remote diagnostics, connection failures, timeouts, and parser failures. Those conditions have different retry behavior. A DNS failure may justify retrying. Exit 2 from a usage error usually does not. Invalid JSON on stdout requires preserving the original bytes so a developer can decide whether the command or the parser was wrong.

A result contract should preserve facts before interpretation

A durable result object contains the raw evidence and makes unknown states explicit. It should not force every caller to reverse engineer a formatted transcript.

This contract is intentionally boring:

{
  "stdout": {"encoding": "base64", "data": "Li4u", "truncated": false},
  "stderr": {"encoding": "base64", "data": "Li4u", "truncated": false},
  "events": [
    {"seq": 1, "stream": "stdout", "offset": 0, "length": 48},
    {"seq": 2, "stream": "stderr", "offset": 0, "length": 19}
  ],
  "termination": {
    "kind": "exit",
    "exit_status": 0,
    "exit_signal": null,
    "core_dumped": null
  },
  "transport_error": null,
  "started_at": "2026-07-24T10:20:30.123Z",
  "finished_at": "2026-07-24T10:20:31.456Z"
}

The two stream objects are authoritative content. Each event references a byte range rather than copying text, so a viewer can build a transcript without duplicating the payload. seq means capture observation order only. It does not claim that remote writes happened in that exact order.

The termination.kind field should cover at least exit, signal, timeout, cancelled, transport_error, and unknown. Use nullable fields rather than magic codes. An absent SSH exit status is not zero, and a local timeout is not exit 124 unless a shell or timeout utility actually produced 124 on the remote host.

Include truncation per stream. A global truncated flag cannot tell a parser whether it still has complete JSON on stdout or merely lost the end of verbose stderr. Record captured byte counts and discarded byte counts when you know them. If you retain only a prefix and suffix, model those segments rather than joining them as though the missing middle never existed.

Timestamps help with latency and investigation, but do not use wall-clock time to order chunks. Clocks can jump, and two concurrent readers can receive the same timestamp at your chosen resolution. Assign the sequence counter at a single serialization point. Keep a monotonic duration separately if your runtime provides one.

Version the contract before clients depend on it. Adding fields is usually safe, but changing events.seq from arrival order to display order is a semantic break even if the JSON shape stays the same.

Cross-stream order has a hard limit

You can preserve the order in which your SSH stack observed channel messages, but you usually cannot prove the order of the remote program's writes across stdout and stderr. That limit should appear in the data model and the interface copy.

Within one stream, bytes stay ordered. Across two streams, buffering intervenes at several layers: the remote language runtime, libc, pipes, the SSH server, transport packets, the client library, and your own reader tasks. Stdout may be block buffered when it is not attached to a terminal, while stderr may flush sooner. A later stderr write can therefore become visible before an earlier stdout write.

RFC 4254 preserves the channel message sequence that the SSH implementation sends. That is useful, and a library callback that exposes those messages can assign a faithful receive sequence. Once a library splits data into independent stdout and stderr readers, two goroutines or asynchronous callbacks race to report readiness. The order in which your scheduler runs them is an observation of local delivery, not a reconstruction of remote source code order.

This small command demonstrates why a test must not demand one universal merged transcript:

sh -c 'printf "out-1\n"; printf "err-1\n" >&2; printf "out-2\n"; printf "err-2\n" >&2'

A terminal often displays the apparent source order. Redirect both descriptors to one file with >all.log 2>&1, and the shell points them at the same destination, which gives you one kernel-managed write path for that process. Capture them through separate pipes, and the observer can receive chunks in another order. Add a language runtime with buffering and the gap grows.

If exact cross-stream chronology is a requirement, change the producer contract. Have the remote program write structured records with its own sequence number to one stream, or direct both descriptors to one remote sink before SSH sees them. That buys a defined order by giving up independent streams at the producer. A generic SSH client cannot manufacture the missing fact afterward.

Audit text should say observed sequence, not execution sequence. That phrase is not legal padding. It keeps an investigator from reading causality into scheduler timing.

Chunk boundaries are transport artifacts

A read callback is not a line, a record, or one remote write call. Parsers that assume otherwise work in tests and fail under load.

One call may arrive as several chunks. Several writes may arrive in one chunk. A UTF-8 code point, an ANSI escape sequence, or a JSON token may cross a boundary. The same command can produce different chunking on the next run without changing its output.

Build the capture layer around byte append operations. For each stream, append the chunk to its buffer or spool file and record the resulting offset and length. If the library exposes messages serially, assign seq there. If it exposes independent readers, send chunk notices to one collector and document that its sequence reflects collector receipt.

Line splitting belongs in a derived view. Maintain one incremental decoder and one unfinished-line buffer per stream. Never share a line buffer between stdout and stderr, because an unterminated stdout fragment followed by a stderr line must not become a synthetic line. When the stream closes, expose the final partial line instead of silently dropping it.

JSON parsing should normally wait until stdout reaches end of stream and the command termination is known. A streaming JSON protocol is different: it needs explicit framing such as newline-delimited JSON, a length prefix, or a documented incremental grammar. Guessing record boundaries from chunks is not streaming; it is a race.

Binary output needs an explicit path too. Base64 inside JSON is simple and portable, though it adds size. A blob reference can work for large results if the audit system guarantees retention and integrity. Do not decode with replacement characters and discard the originals. The replacement hides whether corruption came from the remote tool, the transport adapter, or your viewer.

Limits must apply while reading, not after buffering everything in memory. Continue draining both streams even if one exceeds its retention limit, or the remote process can block on a full pipe. Store the permitted prefix, suffix, or external spool, count discarded bytes, and keep reading until closure or cancellation.

A pseudo-terminal trades structure for terminal behavior

Do not request a pseudo-terminal for a command whose stdout will be parsed. A PTY is useful for a human session, but it changes the program's environment and commonly sends stdout and stderr through the same terminal device before the SSH client can preserve their identity.

Programs inspect whether a descriptor refers to a terminal. They may enable color, draw progress with carriage returns, wrap lines to the reported width, prompt for input, or switch from block buffering to line buffering. The bytes captured with a PTY can therefore differ from the bytes produced by the same command without one. This is observable behavior, not a display option.

The OpenSSH client reflects the distinction in its options: -T disables pseudo-terminal allocation, while -t requests it and repeated -t can force it. Automation should default to no PTY. Ask for one only when the remote program requires terminal semantics and the result contract explicitly says that stream separation is unavailable.

A PTY does not make ordering more truthful. It can give you one terminal byte stream, so the displayed order becomes defined at that terminal boundary, but the program and its libraries may buffer differently because they detected the terminal. You traded separate evidence for interactive behavior; you did not discover the chronology of an execution without a PTY.

This distinction explains a stubborn class of bugs. A developer tests a command by typing it in a shell and sees clean, colored progress in a sensible order. The agent runs the same text without a PTY, stdout becomes block buffered, stderr appears first, and the parser receives output without control codes later. Someone then forces a PTY to make the transcript resemble the manual test, and JSON parsing starts failing because color codes or prompts enter the stream.

Treat interactive and structured execution as different modes in the API. Structured mode should promise separate streams and stable capture behavior without terminal emulation. Interactive mode should return a terminal transcript, terminal dimensions, and an explicit indication that original stdout and stderr identity was not retained. A Boolean pty: true buried in request options is not enough if the response looks identical to a structured result.

Remote startup files add another wrinkle. RFC 4254 warns that shell initialization can produce spurious output when a subsystem starts and recommends a recognizable marker for protocols that need to distinguish it. The same lesson applies to command adapters: invoke the narrowest executable path you control, avoid unnecessary interactive shells, and treat unexpected leading bytes as evidence rather than silently stripping anything that looks like a banner.

If a command truly needs a password prompt or terminal control, do not pretend its transcript is parser-ready. Give the agent a specialized interaction tool with bounded inputs and a transcript designed for terminal semantics. Keeping that path separate protects the simpler guarantee that ordinary SSH actions return faithful stdout, faithful stderr, and a terminal outcome.

Exit status is part of the result, not an exception string

RFC 4254 defines an exit-status channel request and a separate exit-signal form. It recommends returning status, but it also permits clients to ignore it. Your API therefore needs an explicit unknown outcome instead of assuming success when no status arrived.

A zero status usually indicates success, not certainty. RFC 4254 uses that qualified wording because command conventions live above the transport. Still, the status is the primary generic signal available. Preserve the unsigned value the protocol provides before mapping it into a host language's process conventions.

Signal termination is not a negative exit status. Store the signal name, the core-dump flag when supplied, and the remote explanatory message separately. If a consumer wants a shell-like number such as 128 plus a signal value, it can derive one for display. The audit record should keep the SSH facts.

Distinguish these outcomes in code and UI:

• The remote command returned a status.
• The remote side reported signal termination.
• The channel closed without either report.
• The client failed before command start was confirmed.
• The connection failed after partial output arrived.

That fourth case must not masquerade as remote exit 255 just because the OpenSSH command-line client often uses 255 for its own errors. A library transport error has its own type. If you invoke the ssh executable as a subprocess, 255 is all the wrapper may know, so preserve its local stderr and label the boundary honestly.

Completion also means all output has been drained. The Go os/exec documentation warns that calling Wait before reads from StdoutPipe or StderrPipe finish is incorrect. Node.js draws a similar boundary: its exit event can occur while stdio remains open, whereas close follows stream closure. These manuals describe local subprocesses, but the design lesson applies directly to an SSH helper. Publish the final result only after termination is known and both output readers have reached their terminal state.

Timeout and cancellation deserve fields of their own. Record who initiated cancellation when your system knows, whether a signal was requested, and whether the channel actually closed. Do not declare timed_out: true and discard a later remote exit report; both events may matter during an investigation.

Parsers should consume stdout and retain everything else

A command-specific parser should receive stdout bytes, termination, and content metadata. It should not receive a blended transcript and guess which lines are diagnostics.

Suppose an agent runs a remote inventory command that promises JSON on stdout. The adapter should first check that the SSH operation reached a known termination, then apply the command's status policy, then decode and parse stdout. Stderr remains attached to the result as supporting evidence. A warning does not enter the JSON decoder, and a parse error does not erase the warning.

Return parser failure alongside the command result, not instead of it. A useful error can say that stdout byte 418 was invalid while retaining the original stdout, stderr, exit status, and truncation flags. That bundle lets the agent decide whether to repair an invocation, retry with a stable locale, or hand the exact evidence to a person.

Avoid convenience APIs named CombinedOutput in a structured agent path. The Go manual describes exactly what that method does: it returns combined standard output and standard error. It is handy for a one-off diagnostic command and wrong for a reusable result contract because the lost labels cannot be inferred later.

Text commands need command-specific choices too. A parser may treat stdout as newline-delimited records while presenting stderr as plain diagnostic text. Another may accept a zero status plus an empty stdout as a valid empty result. Put these rules beside the command definition, with tests, rather than embedding them in the transport.

Prompt construction should use structured fields. Tell the model exit status: 2, provide stdout and stderr in separately labeled blocks, and state when content was truncated. Do not concatenate untrusted remote output into instructions without boundaries. Output can contain text that resembles a prompt, so treat it as data and escape it for the container format you use.

An agent should not decide success from prose. Give it a machine field such as termination.kind and exit_status, then let prose explain. This reduces token use and stops a warning that contains the word error from overriding a successful status.

Audit views need two honest renderings

The audit record and the human transcript have different jobs. The record preserves bytes and metadata; the transcript helps someone read them.

A useful call view starts with a status strip: command, host identity, start and finish times, termination kind, exit status or signal, byte counts, and truncation. Below it, offer separate stdout and stderr tabs as the authoritative views. A combined tab can interleave event ranges by observed sequence, with a persistent stream label on every row.

Do not encode stream identity with color alone. Use text labels and provide a copy action for each original stream. Copying the combined view should either include explicit labels or warn that it is a rendering, because pasted output without provenance recreates the original problem.

Long lines, carriage returns, and terminal control codes require cautious rendering. Escape control characters by default. A progress bar that repeatedly writes \r should not overwrite older audit content as if the viewer were a terminal. Offer terminal emulation only as an optional derived view, and keep the raw representation accessible.

Search should return the stream, byte offset, and event sequence with every match. Filtering to stderr must not change sequence numbers. If content is truncated, place a visible gap marker where bytes are missing and show the recorded count. Never make the prefix and suffix touch as though they were adjacent in the source.

A timeline can place termination after the last observed chunk, but only when capture confirms both readers closed before finalization. If the connection broke, show the last output event, the transport failure, and an unknown remote outcome. Reducing that to a red failed badge erases the difference between a program failure and loss of evidence.

Sallyport routes SSH actions through its stateless sp-ssh helper and records individual calls in its Activity journal, so this separation belongs at the helper result boundary before any agent or audit view formats the call. The useful product behavior here is not a clever transcript; it is retaining enough evidence that agents and people can reach their own conclusions.

Test the failure shapes, not one happy transcript

A parser test suite should vary chunking, stream timing, termination, encoding, and retention limits independently. Snapshotting one merged string tests your formatter and little else.

Start with a fake channel source that emits protocol-level events under your control. Feed one stdout payload in every possible split point. Then do the same for a multibyte UTF-8 sample, an ANSI sequence, and a final line without \n. The stored bytes must remain identical across every split.

Interleave stdout and stderr events with known sequence numbers and confirm that separate buffers, range offsets, and the combined view agree. For independent-reader implementations, inject scheduling delays and assert only per-stream byte order plus collector observation order. A test that insists on the producer's source-code order is asserting a guarantee the system does not have.

Cover termination combinations that ordinary fixtures miss: zero with stderr, nonzero with empty stderr, signal with partial stdout, channel close without status, transport failure after both streams produced data, timeout followed by a late close, and cancellation before start confirmation. Each should produce a distinct structured outcome.

Put small limits on one stream at a time. Verify that stdout truncation does not mark stderr truncated, discarded byte counts are correct, readers continue draining, and the final status still arrives. Then fill both streams concurrently. This catches the classic deadlock where code drains stdout fully before it begins reading stderr.

Property tests work well for the byte invariants. Generate arbitrary byte slices and chunk boundaries, pass them through the collector, and require that concatenating retained event ranges reproduces the retained stream content. Generate event schedules separately from stream contents so the test never confuses chunking with meaning.

Finally, test every export. JSON must preserve null versus zero. The text transcript must label streams. Redaction must not shift stored offsets without recording a mapping or producing a separate derived artifact. An audit format earns trust when awkward failures remain awkward and visible, rather than being normalized into a tidy but false story.

Keep the raw streams, label the observed order, and wait for both drainage and termination before publishing a result. Once a flattened string enters an agent message or an audit log, the missing distinctions cannot be recovered, and every later layer has to guess.