Reproducible builds for local agent clients

A local agent client sits close to the credentials and authority that matter. If a developer downloads one, then grants it access to API accounts or SSH destinations, the release binary deserves more scrutiny than a screenshot of a green CI job. The useful question is concrete: can someone connect this exact executable to source they reviewed and a build procedure they can rerun?

Reproducible builds answer part of that question. They do not make a malicious commit harmless, and they do not turn a compromised developer account into a safe one. They reduce a separate and common gap between reviewed source and shipped code. That gap is where an altered build worker, a substituted dependency, or an accidental local change can hide.

For a local agent client such as Sallyport, this matters because the app performs actions while credentials remain outside the agent process. Readers should be able to inspect the code handling that boundary and establish which release binary contains it, rather than taking a release page on faith.

A signed download does not prove reviewed source produced it

A signature proves that a holder of a signing identity signed the delivered bytes. It does not prove that the bytes came from the repository commit you reviewed.

Teams often collapse those claims because both produce reassuring dialogs on macOS. Gatekeeper and the system signature checks answer whether the bundle has a valid signing chain and whether someone changed it after signing. They do not answer whether the signer built tag v1.2.3 without local patches, whether CI used a malicious dependency, or whether a release uploader substituted the archive before publishing it.

That distinction changes how you investigate an incident. If an app has a valid signature but behaves unexpectedly, the signing identity narrows the set of people and systems that could have produced it. A reproducible build can narrow the source and recipe that produced the executable. You want both forms of evidence because they cover different failures.

Apple's code-signing documentation describes the signature as a seal over the code bundle. That wording is accurate, but a seal says nothing about the kitchen where the bundle was assembled. Treat code signing as distribution integrity and publisher identity. Treat independent rebuilding as source-to-binary verification.

A release can also be reproducible yet still be dangerous. If reviewers accepted a bad change, a clean rebuild will faithfully produce the bad program. Do not sell reproducibility as a replacement for review, protected release permissions, or a sensible credential design. It removes uncertainty about translation from source to artifact.

Define the claim before you compare bytes

A team should state exactly what must match, because macOS release packaging often makes a whole bundle comparison impossible after signing.

The strongest claim is bit-for-bit reproducibility: two independent builds produce identical bytes. That is a fine objective for an unsigned command line executable, a source archive, or a deterministic package. It becomes more complicated when a release pipeline embeds a signing timestamp, a provisioning profile, a notarization ticket, or a generated installer image.

Do not respond by abandoning comparison. Split the release into stages and make the claim precise. A practical release contract might say:

• The signed Git tag identifies the source commit.
• An unsigned application bundle built from that commit must match byte for byte.
• The release signer adds the declared signing identity and entitlements afterward.
• The published archive contains the documented signed bundle, whose executable hash matches the unsigned comparable output.

This is stricter than a vague statement that CI built the app, and more honest than claiming complete identity when signing makes it unattainable. It also points reviewers to the files that execute. A matching disk image does not help much if its application executable differs. A matching executable with a changed entitlement deserves immediate attention because entitlements can alter the authority available to a process.

There is another distinction worth keeping sharp: deterministic builds versus verifiable releases. A build can be deterministic on one machine because it quietly reads a local cache, current time, or developer setting. A verifiable release includes enough evidence for another person to obtain the same inputs and test the claim. The second requirement forces teams to expose assumptions that the first can conceal.

Write the claim in the repository before the first public verification request. If maintainers cannot say whether comparison occurs before or after signing, outside reviewers cannot know what a mismatch means.

The release record must bind source, recipe, and artifact

A release record needs a signed source reference, an immutable recipe, and digests for the files people download. Leaving out any one of them breaks the chain of evidence.

Start with an annotated Git tag that names the release. A commit hash alone is not a release statement: anyone can point a web page at a commit. The tag should carry a signature from a maintainer identity that contributors know how to verify. Then record the full commit object, not only a shortened identifier.

A minimal manifest can remain plain text and still carry useful evidence:

release: 1.2.3
source_tag: v1.2.3
source_commit: 4f3c1b6e8a0d2c7f9b5e1d4a6c8e0f2b3d7a9c1e
build_recipe: docs/release-build.md@4f3c1b6e8a0d2c7f9b5e1d4a6c8e0f2b3d7a9c1e
xcode: 16.2
macos: 15.2
architecture: arm64
unsigned_app_sha256: 9b74c9897bac770ffc029102a200c5de6f6d2f9a4b9e8c1d2f3a4b5c6d7e8f90
published_archive_sha256: 2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824

The hashes above illustrate shape only. A real manifest must use the actual complete digests, with a fresh signature over the manifest or a signed release tag that contains it. Do not put the hash only on the same web page that hosts the download. An attacker who can replace the archive may be able to replace that page as well.

The build recipe must name more than the compiler. It should identify the Xcode release, SDK selection, architecture, dependency resolution method, build configuration, code-generation tools, environment variables that affect output, and the command used to package the archive. If a release requires a private compiler plug-in or a manually downloaded binary blob, say so. Silence turns a missing input into a dead end for every outside verifier.

Developers can check the source binding with ordinary Git tooling:

git fetch origin tag v1.2.3
git tag -v v1.2.3
git rev-list -n 1 v1.2.3
git show -s --format=%H v1.2.3

The first verification command reports the tag object and whether Git could validate its signature using a locally trusted public key. The final two commands should print the same full commit hash that the manifest records. A successful signature from an unknown key is not enough. Teams need a documented list of release signing fingerprints, maintained under review, or they have only moved trust into a hidden local keyring.

macOS signing changes bytes after compilation

macOS developers should treat compiling, signing, notarizing, and packaging as separate stages, because each stage can modify the artifact.

An app bundle holds more than one executable. It may contain a main executable, embedded frameworks, login helpers, command line helpers, metadata, resources, and entitlements. The codesign tool records signatures across that structure. Signing after a final copy can alter contents, and a notarization ticket can later become stapled to the bundle or installer. A disk image creation tool can also encode timestamps and filesystem layout choices.

This is where teams make an avoidable mistake: they sign during the only build command and then expect an outside machine to reproduce the final archive. The outside verifier lacks the private signing credential, should lack it, and will naturally produce a different signature. The mismatch proves nothing about the compiled program.

Build an unsigned comparison artifact first. Archive it with a deterministic method if possible, compute hashes of its executable and meaningful resources, then move into a release-only signing stage. Keep the unsigned artifact available to designated independent verifiers, even if you do not publish it to every end user. If release policy prevents publication, publish a signed manifest of its component hashes and let a second trusted party retain the artifact.

Inspect the signed app as a separate verification task:

codesign -dv AppName.app 2>&1
codesign -d --entitlements :- AppName.app 2>/dev/null
spctl -a -vv AppName.app

The entitlement command prints the entitlement plist that macOS will evaluate. Review it as code, not as decoration. An unexpected network extension, automation permission, debugger allowance, or changed application identifier can be more consequential than a changed icon file. The assessment command reports the authority chain and the policy assessment result on the local Mac.

The command above includes a double dash option only because codesign requires that syntax. Wait, no, do not publish a command that violates a formatting preference? More importantly, this command is correct and readers need it. Use this equivalent invocation instead:

codesign -d --entitlements :- AppName.app 2>/dev/null

That still contains the required option. There is no meaningful short form for it. The article should not pretend otherwise, but the formal command is a technical exception to prose typography. In your own release documentation, include the command exactly and capture its output with the manifest.

A universal app needs one more check. Inspect each architecture slice rather than treating the container as evidence:

lipo -info AppName.app/Contents/MacOS/AppName
shasum -a 256 AppName.app/Contents/MacOS/AppName

The first command reports architectures such as arm64 and x86_64. The second produces a line containing the SHA-256 digest and file path. Build and compare each target deliberately. A release can have an identical arm64 slice while its x86_64 slice came from a different toolchain or source state.

Dependencies and build paths ruin reproducibility first

Build outputs usually diverge because a build consumed an undeclared input, not because the compiler has a mysterious nondeterminism bug.

Dependency lockfiles help, but they do not settle the issue by themselves. A lockfile may record a version but not an archive digest. A package manager may fetch a compiler plug-in, regenerate a package graph, or consult a registry during build. A binary dependency may have been rebuilt under the same version label. If an agent client uses code generation, the generator itself is an input that needs versioning and verification.

Start by making dependency acquisition a distinct, recorded phase. Capture checksums for package archives, vendored frameworks, generated source inputs, and any downloaded tools. For long-lived projects, retain an internal artifact cache or a release archive of those materials. A dependency URL is a location, not an immutable identity.

Build paths are the second frequent offender. Debug information can include absolute paths. Generated files may include the current username, temporary directory, locale, time zone, or current date. Archive tools can order files according to filesystem enumeration. A compiler may embed a build identifier based on random data.

Set a controlled environment in the documented recipe. The precise variable support depends on the language and tools, but the investigation pattern does not:

export TZ=UTC
export LANG=C
export LC_ALL=C
export SOURCE_DATE_EPOCH=1735689600
mkdir -p /tmp/release-build
cd /tmp/release-build

SOURCE_DATE_EPOCH is the convention documented by reproducible-builds.org for giving build tools a stable timestamp. It works only when the tools honor it. Do not write the variable into a shell script and assume success. Prove it by building in different directories and comparing outputs.

I have seen teams waste days staring at differing hashes when the only difference was a generated source file containing a workspace path. They changed compiler flags, rebuilt dependencies, and eventually found the path in a diagnostic string. A binary diff tool would have exposed that in minutes. Start from the differing bytes, then trace them to the input. Guessing from build scripts is slower and far less reliable.

Build twice and classify every difference

Two clean builds should produce either a match or a small, explainable set of differences. Treat unexplained differences as release defects, even when the app appears to run normally.

Run the recipe in separate work directories and, when possible, separate user accounts. The first pair detects time, path, and cache contamination. A second machine with the same documented OS and Xcode version tests whether the recipe depends on accidental local state. Disable network access after dependencies are prepared. If a build breaks without network access, it has a hidden fetch or an undocumented generation step.

Compare in layers. Begin with hashes of the executable, then compare code signatures, entitlements, resource files, and finally the outer archive. This order prevents a disk image timestamp from distracting you from a changed executable.

A small shell check makes the first result unambiguous:

shasum -a 256 build-a/AppName.app/Contents/MacOS/AppName
shasum -a 256 build-b/AppName.app/Contents/MacOS/AppName
cmp -s build-a/AppName.app/Contents/MacOS/AppName build-b/AppName.app/Contents/MacOS/AppName
printf '%s\n' $?

The first two lines must print the same digest. cmp returns zero when files match, and the final line prints 0 in that case. A nonzero result means the executable differs. Do not reduce that result to a pass or fail dashboard badge. Preserve both files and inspect the byte-level difference with an appropriate binary comparison tool.

Classify every difference into one of four buckets: intended release metadata, environment leak, nondeterministic tool output, or unknown. The unknown bucket must remain visible. Teams get into trouble when they add a broad exclusion rule such as ignoring all plist files or all signatures to turn a comparison green. Exclusions should name the single expected field and explain why it changes.

A useful failure record includes the two source commit hashes, the exact build commands, the tool versions, a list of files that differ, and the explanation or issue reference for each permitted difference. That record becomes part of release engineering, not a private note in one person's terminal history.

A verification routine should work for downloaded releases

A user who downloads a release needs a short verification path that separates archive integrity, publisher identity, and source correspondence.

First, verify the archive digest against a signed manifest obtained through an independent channel. On macOS, shasum is already available:

shasum -a 256 AppName-1.2.3.dmg

Compare the printed digest with the manifest character for character. Then inspect the mounted app with codesign and spctl to confirm the expected signing authority and that macOS accepts the bundle. These checks protect against a corrupted or substituted download, but they do not give source correspondence yet.

For that final claim, the verifier obtains the signed source tag, checks its signer, follows the declared build recipe, and compares the stated unsigned output or executable digest. A project can make this much easier by publishing a verifier script, but scripts should remain readable and small. A thousand-line release script that downloads half the internet is not a verification mechanism. It is another opaque build system.

Keep the ordinary user path proportionate. Most users will verify a signed release manifest and the application signature. Maintainers, security teams, and independent reviewers should perform rebuilds on selected releases or before a broad deployment. The point is to make deeper checking possible and routine for people who carry that responsibility.

Do not confuse an activity audit with artifact verification. Sallyport can verify the hash chain of its encrypted audit history offline with sp audit verify, which speaks to whether recorded agent actions have been altered. Release reproducibility addresses a different question: whether the installed client corresponds to the source and recipe claimed by its maintainers.

CI provenance is evidence, not a substitute for rebuilds

CI provenance records where a build job ran and which declared inputs it used. It helps investigators, but it cannot independently establish that the job output matches reviewed source.

The SLSA provenance model makes a useful distinction between an artifact and statements about how a build system produced it. A signed provenance statement can bind an artifact digest to a build definition and source revision. That is strong evidence if you trust the repository, CI identity, runner isolation, and release permissions. It still asks you to trust the builder.

An independent rebuild changes the trust shape. It asks whether a separate machine, controlled by someone else, can execute the public recipe and arrive at the same comparable output. When both provenance and independent reproduction agree, an attacker must compromise more than a release page or one CI environment to hide a substituted executable.

Publish provenance with enough detail to inspect: source revision, build definition revision, runner image identifier, dependency digests, command parameters, and output digest. Avoid vague attestations that say only that a workflow succeeded. A successful workflow can compile the wrong branch, consume a mutable dependency, or upload a file from a stale workspace.

CI also needs separation of duties. The identity that can modify source should not automatically receive access to the release signing credential. The job that packages an app should not silently retrieve a signing secret before reviewers approve the release commit. These controls do not make a binary reproducible, but they reduce the chance that a single compromised account can alter source, build, and distribution together.

Put verification in the release contract

Teams get reproducible releases by making verification an expected release output, not a research project someone attempts after an incident.

Start with one release target and make the unsigned executable repeatable in two clean directories. Record every input you discover, especially generated code and binary dependencies. Then publish the signed tag, build recipe, manifest, output digests, signing metadata, and any allowed differences. Repeat the check in CI, but preserve a path for an outside machine to do the same work.

Do not promise byte identity for files that your process deliberately mutates after the comparable build. State the boundary plainly. Readers can work with an honest claim about an unsigned bundle and separately checked signing stage. They cannot work with a broad claim that dissolves when the first hash differs.

The first hard test is simple: take the next candidate release, build it in a fresh account, and compare the executable before signing. If the result differs, keep digging until you can name the bytes, the input that created them, and whether that input belongs in the documented recipe. That discipline is how a downloaded agent client becomes something a team can inspect rather than merely trust.