Local agent action gateway health checks

A local action gateway is healthy only when it can refuse the wrong action, perform the right controlled action, and leave evidence that survives review. A green process monitor proves almost none of that. It tells you that something has a PID. It does not tell you whether the vault is locked, whether an agent can reach the gateway, whether injected credentials still work, or whether the audit trail recorded the call.

That distinction matters with Sallyport because the app, the vault gate, the authorization decision, the external action, and the encrypted audit log are separate points of failure. Treating them as one generic "health" signal creates the worst kind of monitor: one that stays green while the control you rely on has stopped working.

Build this as a small acceptance suite, not as a pile of port checks. The suite should use harmless canary targets, return non-secret evidence, and classify a locked vault correctly. If a person must unlock the vault with Touch ID, your monitor should report that state plainly instead of trying to bypass it.

A running app is only the first condition

An availability probe should answer one narrow question: can the local Mac start and keep the gateway app running? It should not pretend that a running process proves external actions work.

For a menu-bar app, start with a local check that a supervisor can run without credentials. The exact process name and install location can vary by release, so keep those values in one local configuration file rather than scattering them through scripts. A basic shell probe can look like this:

#!/bin/sh
set -eu

APP_NAME="Sallyport"

if ! pgrep -x "$APP_NAME" >/dev/null 2>&1; then
  open -a "$APP_NAME"
  sleep 2
fi

if pgrep -x "$APP_NAME" >/dev/null 2>&1; then
  printf 'app=ready\n'
  exit 0
fi

printf 'app=unavailable\n' >&2
exit 2

This has an intentionally modest claim. open -a asks macOS to launch an application, and pgrep observes a process after that request. It does not prove that the app finished initialization, that its vault can answer a request, or that its MCP shim can accept a client. Apple documents Launch Services as the system interface used to launch and activate applications, which is why launching through the operating system is better than hard-coding an application bundle path.

Keep the output structured and dull. app=ready is enough for a scheduler. Do not dump ps output into a central log where command arguments, usernames, or unrelated process details become permanent clutter.

A process can exist while it is wedged. It can also be absent because the Mac is asleep, logged out, or restarting after an update. Your alerting rule needs a grace period for those expected conditions. A health check that pages someone every time a laptop closes its lid will be disabled, and then it will not help during a real failure.

The useful availability signal is therefore local and bounded: the app started, remained present for a short settling interval, and an MCP client can begin a session. The last part belongs in a separate probe because it exercises a different boundary.

The vault state must be an explicit result

A locked vault is healthy when it denies actions. Calling that state an outage confuses security behavior with service failure.

Sallyport's vault gate is absolute: while locked, every action is denied. On supported macOS hardware, the vault gate uses Secure Enclave and Touch ID. A check must preserve that rule. Do not write a script that feeds passwords to a UI, stores a biometric bypass token, or treats an unlocked desktop session as proof that the vault should be unlocked.

Use four result states instead of a Boolean:

• ready: the app is available, the vault is unlocked, and controlled checks may run.
• locked: the app is available, but the vault correctly denies actions.
• denied-unexpectedly: the vault is unlocked, yet the intended canary action was denied.
• unavailable: the app or its local MCP path cannot answer.

That vocabulary prevents a common operational mistake. Teams often schedule a credentialed probe at night, see failures after the screen locks, and weaken the system until it can unlock itself. They have not fixed monitoring. They have removed the human decision that the vault was built to require.

The practical pattern is a two-part run. An unattended job records app availability and locked-vault denial. A person, or a controlled workstation session that already has human approval, initiates the credentialed checks after unlocking. Record the reason in the run output:

{
  "run_id": "hc-2026-07-22T141501Z-8f29",
  "app": "ready",
  "vault": "locked",
  "http": "skipped",
  "ssh": "skipped",
  "audit": "verified",
  "reason": "credentialed checks require an unlocked vault"
}

The run ID is not a secret. It gives operators something stable to compare with the activity record later. Do not use a username, machine serial number, endpoint URL, or credential label as the run ID.

Per-session authorization and per-call approval also need their own treatment. A new agent process may require session approval, and a credential may require approval for each use. That is expected behavior, not a flaky test. Your runner should state whether it is designed for an approved session or whether a human will approve each call. A silent timeout tells the next investigator nothing.

Use canary targets that prove credential injection

An HTTP check should call an endpoint built to validate a specific canary credential and return a fixed, non-secret result. Testing a public URL only proves that the network works. It says nothing about whether the gateway selected the intended credential, inserted it into the right header, or kept it away from the agent.

Set up a tiny service that accepts one path, one method, and one credential form. Store the expected canary secret on the service side. Store the same canary secret in the gateway vault. The client that triggers the action should never receive the secret, and the service should never echo it.

The response contract can be this small:

{
  "check": "agent-gateway-http",
  "result": "ok",
  "request_id": "7d7a0f3c"
}

The service should return 401 when the credential is absent or wrong, 405 for the wrong method, and 200 only when the expected credential arrived. Generate request_id at the service and keep it opaque. Do not derive it from the authorization header or any part of the incoming request.

Use a dedicated route such as /agent-gateway-canary. Do not bolt the check onto an existing production endpoint. Production endpoints accumulate behavior over time: rate limits, redirects, content negotiation, caching rules, billing side effects, and permission changes. A canary route can stay intentionally boring.

RFC 9110 defines request method semantics and classifies GET, HEAD, OPTIONS, and TRACE as safe methods, but safe in HTTP means the requested action should not change the resource's intended state. It does not mean the call is harmless for your account, logs, quotas, or downstream behavior. An API can record a GET, charge a request, or trigger a poor implementation's side effect. Build a route whose server-side behavior you can inspect, rather than trusting a familiar verb.

A frequent bad recommendation is to use curl with a production API token as the health check. It is popular because it takes one line. It is wrong because shell history, process inspection, CI logs, and error output create too many places for a bearer token to appear. It also bypasses the behavior you need to test if the gateway normally injects credentials itself.

Your runner should invoke the gateway through the same MCP route that an agent uses. Do not invent a backdoor HTTP client for the check. Keep the transport-specific invocation behind a local adapter because tool names and request shapes can change. The adapter takes a logical action, asks the MCP-connected gateway to perform it, and emits only the normalized result.

{
  "action": "http_canary",
  "target": "canary-api",
  "method": "POST",
  "path": "/agent-gateway-canary",
  "expected_status": 200,
  "expected_check": "agent-gateway-http"
}

The adapter must redact on failure. It may report http_status=401, transport_error=timeout, or response_schema=invalid. It must not print outgoing headers, a request body, a full URL with query parameters, or the raw response unless you have reviewed that data as safe.

An HTTP success must prove the intended action

A 200 by itself is weak evidence. The check must validate the service response, the method, and the target identity so that a redirect, proxy page, or stale fixture does not turn into a false pass.

Make the canary response identify the test without identifying a credential. Compare a few exact fields:

status=200
check=agent-gateway-http
result=ok
request_id=7d7a0f3c

The checker should accept any syntactically valid request_id, then save it with the run ID. It should reject a missing field, a body that claims success but has a non-2xx status, and an unexpected content type. A captive portal, a corporate proxy error page, or a misrouted DNS record often returns a valid HTTP response. That is transport success, not action success.

The curl documentation makes a related point: without --fail or --fail-with-body, curl does not treat an HTTP status such as 404 or 401 as a command failure. That behavior is correct for a general transfer client, but it catches people who write monitors that look only at curl's exit status. If your adapter uses curl internally, capture both its process result and the HTTP status, then decide success from your explicit contract.

Do not let a check follow redirects automatically unless redirects are part of the intended endpoint design. A redirect can send your canary call to a login page that returns 200, or to another host that you did not mean to contact. Pin the HTTPS origin in configuration, validate the certificate through the normal client stack, and record the final peer identity only if that record cannot reveal sensitive network details.

Timeouts need separate labels. A DNS failure, TCP refusal, TLS validation failure, gateway denial, upstream 401, upstream 500, and response mismatch have different owners. If the runner calls all of them http=failed, you will waste the first ten minutes of every incident finding out where the request stopped.

A useful failure record looks like this:

{
  "run_id": "hc-2026-07-22T141501Z-8f29",
  "check": "http_canary",
  "outcome": "failed",
  "stage": "upstream_response",
  "http_status": 401,
  "request_id": null,
  "secret_material": "redacted"
}

The secret_material line is a reminder for humans reading the record, not proof that redaction worked. Prove redaction with tests: deliberately make the canary reject the credential, capture the runner's stdout and stderr, and search those files for the test secret. The secret must not appear. Repeat with malformed JSON, a timeout, a TLS failure, and an agent-side tool error. Error paths are where secrets usually escape.

SSH needs a fenced target, not a login shell

An SSH health check should prove authentication and command execution against a dedicated account whose server refuses to run arbitrary commands. A successful connection to a general administrative host proves too much in the wrong direction: it gives the health credential a useful shell.

Create a separate account such as gateway-health on a controlled test host. Give it a forced command in authorized_keys or an equivalent server-side restriction. The forced command should ignore the original command, write a timestamp and opaque run ID to a local audit file, then return a constant response.

A conceptual authorized_keys entry looks like this:

command="/usr/local/libexec/gateway-health",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty ssh-ed25519 AAAA... gateway-health

The public key belongs to the gateway's canary identity. The private key stays in the vault. The AAAA... text is intentionally incomplete here because you must generate your own key material, not copy an example into a production file.

The server command should avoid echoing $SSH_ORIGINAL_COMMAND, environment variables, or authentication details. It can return a fixed shape:

{"check":"agent-gateway-ssh","result":"ok","receipt":"c2b91a"}

If you need a run ID for correlation, pass an opaque token in the submitted command and have the forced command validate a tight format such as hexadecimal characters of a fixed length. Never accept an arbitrary string and write it to a shell command, a filename, or a log line. Better still, let the server generate the receipt and join records by time window during investigation.

The check should fail if the server permits an interactive shell, forwards a port, allocates a TTY, or executes a command other than the forced one. Those are configuration regressions. They deserve a different severity than a routine network timeout because they change the exposure of the canary credential.

Do not reuse the SSH identity used for deployments. Reuse makes the check easier to set up, then turns a monitoring failure into a broad access problem. A health credential should have one job, one target account, and no permission beyond producing the receipt.

Sallyport sends SSH through its bundled stateless sp-ssh helper. Your test should therefore go through the normal gateway action path rather than invoking a local private key with OpenSSH. Otherwise you have only checked the host and the account. You have skipped the credential boundary that needs proof.

Audit verification is a separate assertion

A successful canary action and a valid audit trail are different claims. Verify both.

The activity record tells you that the gateway recorded an individual action. The session record tells you about the agent run and gives you a way to revoke that run. Neither should be treated as a substitute for checking the remote effect, because a request can be logged before an upstream failure. Conversely, a remote system can receive a request while a local logging path fails at a bad moment. You need correlation, not wishful thinking.

For each successful HTTP or SSH action, collect three non-secret pieces of evidence:

• the health run ID generated by the runner;
• the opaque receipt or request ID generated by the canary service;
• the local action timestamp, recorded in UTC.

Then inspect the activity journal for the action metadata that is safe to retain, such as channel, action outcome, target alias, and time. Do not expect the journal to contain plaintext credentials. It should not. Do not require it to reproduce the remote response body either. The receipt belongs to the remote canary system, not to the gateway's secret store.

Sallyport projects its Sessions and Activity journals from a write-blind encrypted, hash-chained audit log. Run its offline integrity check as an independent part of the suite:

sp audit verify

A passing command should be recorded as audit=verified; a nonzero result is an integrity incident until proven otherwise. This verification does not need the vault key, which makes it suitable for an unattended local check even while the vault is locked.

Do not reduce audit verification to "a log file exists." File existence catches almost nothing. A truncated file, replaced records, a broken chain, or a writer that stopped after startup can all coexist with a path that looks normal in Finder.

There is another easy error: checking only that the audit chain validates. A chain can validate and still lack the event you expected if the gateway never attempted the action. Pair chain verification with an event-presence assertion after each canary action. Define a reasonable arrival window for local projection and report a delay separately from a missing event. A delayed journal update may need investigation, but it is not identical to a failed write.

Keep the runner outside the trust boundary it checks

The health runner should orchestrate actions and judge evidence. It should not hold credentials, parse vault files, or read secret-bearing application state.

A practical layout has four components:

1. A local scheduler launches a small runner under a dedicated macOS account.
2. The runner checks app presence and invokes a local MCP adapter.
3. The adapter requests named canary actions through the gateway and returns redacted structured results.
4. The runner verifies audit integrity and writes one short report to a protected local directory.

The dedicated account should not have access to the vault's data files. Apple places non-sandboxed macOS application support data under the current user's ~/Library/Application Support directory, but a monitor should not assume that it can or should inspect those files. The check needs behavior, not a copy of the vault's contents.

Keep configuration declarative and free of secrets. This example gives the runner enough information to validate outcomes without disclosing credentials:

checks:
  http_canary:
    target_alias: canary-api
    expected_status: 200
    expected_check: agent-gateway-http
    timeout_seconds: 10
  ssh_canary:
    target_alias: canary-ssh
    expected_check: agent-gateway-ssh
    timeout_seconds: 10
  audit:
    command: sp audit verify
    timeout_seconds: 15

Target aliases matter. An alias is less sensitive than a full URL or host name, and it forces you to make target selection explicit in the gateway configuration. If an operator changes an alias, the action check should show that configuration review is required before it silently begins testing a different place.

Do not give the runner permission to approve a new agent process. Per-session authorization is on by default for a reason. The first call from a new agent process identifies its code-signing authority in the approval card, and approval lasts only for that run. Arrange a long-lived, reviewed health-client process if you need scheduled checks after a person approves it. If the process exits, expect the next run to ask again.

That choice is a little less convenient than a hidden always-approved automation account. It also keeps a new executable from acquiring permission merely because it copied the health check's command line.

A failed check needs a useful diagnosis

Most gateway health failures are ordinary configuration drift. The dangerous part is how quickly people respond by weakening controls instead of locating the broken boundary.

Consider a real-looking sequence. The app is running. The vault is unlocked. The HTTP canary returns 401. The first instinct is to paste the canary token into a shell and call the endpoint directly. Do not do that. It proves that the token works, but it bypasses the vault and creates a new secret leak path.

Work through the evidence in order instead:

1. Confirm that the local MCP adapter reached the gateway and received an action result rather than timing out before submission.
2. Confirm that the target alias still selects the intended stored credential and endpoint configuration.
3. Check the canary service logs for the opaque request ID, method, route, and rejection reason. Do not log its received authorization value.
4. Inspect the activity journal for the matching action attempt and verify the audit chain with sp audit verify.
5. Rotate the canary credential if configuration review shows that its expected value changed or may have been exposed.

This sequence separates four failures that otherwise look identical: an incorrect target mapping, a revoked or rotated secret, a gateway injection failure, and an upstream service configuration change. It also avoids the familiar self-inflicted incident where an operator "tests" a secret in a terminal, then finds it copied into shell history, a scrollback buffer, or a support bundle.

Classify severity by the control that failed. An unavailable app is an availability issue. A locked vault that denies an action is informational unless a scheduled approved check was expected. A canary action that succeeds while the vault reports locked is a security incident. A broken audit chain is also a security incident, even if both canary targets still return success.

Do not hide these categories behind one red or green badge. The operator who sees vault=locked, audit=verified, and http=skipped knows exactly what to do. The operator who sees health=warning starts guessing.

The smallest useful schedule has two lanes

Run non-secret checks frequently, then run credentialed canary actions sparingly and deliberately. The two lanes produce less noise and give you better evidence.

The unattended lane can run whenever the Mac is expected to be awake. It checks that the app is present, that a locked vault refuses action when locked, that the MCP path can return a classified response, and that sp audit verify passes. None of those checks should require extracting a credential or unlocking the vault.

The credentialed lane runs after a human unlocks the vault and approves the health-client session if required. It performs one HTTP canary action, one SSH canary action, confirms the remote receipts, checks for corresponding local activity records, and verifies the audit chain again. Use a modest cadence. Every credentialed action becomes an audit event and a remote service event, so a check every minute creates noise without giving you much more confidence.

Run the credentialed lane after changes to credential configuration, target aliases, agent tooling, macOS permissions, network controls, or the gateway app. That timing catches the changes most likely to break action execution. It also gives a reviewer a clear before-and-after record.

The standard is simple: a health suite must prove the controls you intend to rely on. If it can only prove that an app icon is still in the menu bar, call it an availability probe and stop there. Do not call it proof that autonomous agents can act safely.