Do decoded size limits protect an agent from compressed APIs?

A compressed response is not small because it arrived small. It is small only until your client decodes it, parses it, stores it, logs it, and perhaps hands it to an agent that treats it as evidence. If you enforce a limit only on the bytes received from the network, you have measured the least interesting part of the transaction.

I have seen teams add a sensible-looking 5 MB HTTP limit, celebrate the test, and then discover that a few kilobytes of compressed repeated data could still produce a response large enough to stall a process or contaminate an agent run. Nobody needed an exotic exploit. One endpoint returned a report with a field repeated far more often than expected, compression did its job extremely well, and the client did the dangerous thing: it collected the decoded body before deciding whether it wanted it.

Decoded size limits need to sit on the decoded stream, before a parser, logger, cache, or agent context consumes the body. Keep a separate cap on wire bytes, because that stops slow or unexpectedly large transfers. The two limits address different failures. Confusing them is how a defensive check turns into a false sense of safety.

The wire size tells only part of the story

Content-Length normally describes the HTTP message body as transferred. When the response has Content-Encoding: gzip or Content-Encoding: br, that means the count describes compressed bytes. A 40 KB body can expand into tens or hundreds of megabytes after decoding if the input contains enough repetition. The exact ratio is not the point. Any ratio that exceeds your allocation or context budget is enough to cause trouble.

RFC 9110 treats content coding as a transformation applied to a representation. That wording matters. The representation your application uses is the decoded form, while the transport carries an encoded form. A client that bases its application limit on the transport form has put the guard on the wrong side of the transformation.

Transfer-Encoding complicates the picture further. Chunked transfer has no useful final Content-Length for a client to rely on, and an HTTP/2 or HTTP/3 response does not use chunked transfer in the same way. Even with a header present, a server can send a wrong value. Use the header as an early rejection hint, never as proof that the body is safe.

There are three quantities worth recording for a response: wire bytes read, decoded bytes produced, and bytes retained for the caller. They will sometimes be the same. They often are not. A JSON response may decode to 12 MB, parse into much more memory than 12 MB, and then need trimming to 64 KB before an agent can use it safely.

This distinction also prevents a common argument over a single "maximum response size" setting. One engineer means socket bytes. Another means decoded bytes. A third means the text inserted into a tool result. Give each limit its own name and enforce each one at the boundary it describes.

Decode before any unbounded buffering

The safe order is straightforward: limit the raw response stream, select and initialize the decoder, limit the decoded stream, then parse or retain only what the caller needs. Do not call a convenience method that reads the whole decoded response into a byte slice and check its length afterward. By then the decoder has already spent the memory you were trying to preserve.

For a gzip response in Go, the decoded limiter belongs around the gzip reader. This helper deliberately reads one byte beyond the allowed amount. Without that extra byte, a response whose real size is exactly the limit is indistinguishable from a larger response that was cut off at the limit.

var ErrDecodedBodyTooLarge = errors.New("decoded response exceeds limit")

func readGzipBody(r io.Reader, limit int64) ([]byte, error) {
    zr, err := gzip.NewReader(r)
    if err != nil {
        return nil, err
    }
    defer zr.Close()

    bounded := &io.LimitedReader{R: zr, N: limit + 1}
    body, err := io.ReadAll(bounded)
    if err != nil {
        return nil, err
    }
    if int64(len(body)) > limit {
        return nil, ErrDecodedBodyTooLarge
    }
    return body, nil
}

Apply a raw limit before gzip.NewReader too. That limit does not replace the decoded limit. It stops a peer from sending an enormous compressed stream, and it bounds how much work the client accepts before the decoder has enough input to make progress.

Do not silently truncate and continue. A truncated JSON document usually fails to parse, but a truncated text or line protocol can look plausible. If you keep a preview for diagnostics, mark it as a preview in a field that cannot be mistaken for the complete body. The action itself should fail because the client did not obtain a complete, permitted response.

A decoder may detect a corrupt checksum only when it reaches the end of the stream. When the decoded limit fires, stop reading and reject the response. You do not need to finish verifying an oversized body that you have already decided to discard. When a response remains under the limit, read to EOF and let the decoder report truncation or corruption normally.

A response can exhaust context before it exhausts memory

Memory protection is necessary, but agent tools have another budget: the amount of result text that can safely enter the agent conversation. A 2 MB JSON response may be harmless to a desktop process and still be a terrible tool result. It can crowd out the task, prompt the agent to chase irrelevant records, or force the model to reason over a partial representation that looks complete.

Do not use the context budget as a reason to raise the decoded body limit. They protect different operations. The decoded limit allows the transport and parser to finish safely. The presentation limit controls what the agent receives after successful decoding and, where applicable, structured parsing.

For structured data, choose a projection instead of cutting arbitrary bytes. If an endpoint returns a list of records, retain a bounded number of records and a bounded amount of text per field. Include the total record count only if the parser obtained it without retaining the full list. State that the result was reduced and provide the selection rule, such as "first 50 records ordered by timestamp." That makes the result auditable and prevents an agent from treating a fragment as an exhaustive search.

For plain text, preserve complete lines when possible. A log preview that ends halfway through a line is less useful and can hide the field that explains it. Set a byte budget, read line by line with a line-length cap, and report both the retained count and the reason additional content was omitted. A line reader without its own token size limit simply moves the allocation problem into a different helper.

This is where a lot of "but the model can summarize it" thinking falls apart. The model can summarize data you deliberately selected. It cannot make an unbounded transport safe, and it should not decide how much raw remote output your process allocates before you have checked it.

Content coding must be explicit in the client contract

Treat Content-Encoding as input that selects a decoder, not as decoration around an otherwise identical byte stream. Accept only the encodings your client implements and reject unknown values clearly. If a service returns gzip, use the gzip path. If it returns br, use a Brotli decoder wrapped in the same decoded byte limiter. If it returns no content coding, wrap the raw body in that decoded limiter because raw bytes are also decoded bytes in that case.

Do not accept multiple content codings casually. HTTP permits a list of codings, and the listed transformations have an order. Supporting gzip, br means you must decode in reverse order with a size guard after every expansion stage. A single final limit is weaker than it looks because an intermediate stage can grow large before the final stage reduces it. If your integrations do not require stacked codings, reject them until you have tests and a deliberate implementation.

Automatic decompression deserves inspection. Many HTTP libraries add Accept-Encoding, decode gzip, and hide the change from application code. That is convenient for normal requests, but it can leave the code counting bytes at the wrong layer. Find out whether your response body yields wire bytes or decoded bytes, whether the library removes Content-Encoding, and whether it exposes compressed byte counts. Write a test that proves the behavior for your exact client configuration.

Brotli deserves the same treatment as gzip. It is tempting to write a gzip test because every environment has a gzip command and assume the work is done. A different decoder means different error handling, buffering behavior, and support in dependency versions. The limit must surround the reader returned by each decoder, not sit in a helper that only the gzip route happens to call.

A server can also send a declared encoding with an invalid body. Preserve that as a decoding error, distinct from an oversized decoded response. Operators need to know whether the remote side sent bad data, the configured budget was too low, or the client has no decoder for a declared coding.

Test expansion with files you can inspect

A useful fixture starts with decoded content that you can recognize and measure. Repeated bytes give an obvious expansion case. The following commands create a 32 MiB text body, then make gzip and Brotli versions when the Brotli command is installed.

python3 -c 'open("repeat.txt", "wb").write(b"A" * (32 * 1024 * 1024))'
wc -c repeat.txt
gzip -9 -c repeat.txt > repeat.txt.gz
brotli -f repeat.txt -o repeat.txt.br
wc -c repeat.txt.gz repeat.txt.br

The first wc output should have the shape 33554432 repeat.txt. The compressed files should be dramatically smaller than the source because the input repeats one byte. Do not make a test assert a particular compressed size. Compression versions and settings can change it. Assert the decoded length, the client outcome, and the fact that the client retained no full body after rejecting it.

Build boundary cases from the same source: a decoded body one byte below the configured limit, exactly at the limit, and one byte above it. Exercise each case through a local HTTP handler that sets Content-Encoding correctly. A file decoder test is useful, but it cannot catch a client library that performs automatic decompression before your limit code runs.

Use a second family of fixtures with realistic structure. Generate newline-delimited JSON where every record has a repeated payload field, then serve it under gzip and Brotli. This catches code that handles a byte slice correctly but lets a JSON parser or line scanner build an unbounded array after decoding. Include one record longer than your field or line limit, because attackers do not have to repeat short lines to make a parser allocate.

Keep the fixtures checked in only if their compressed forms are small and their source can be generated during tests. A generator script that emits a known decoded length is easier to review than a mysterious binary blob. Record the intended decoded size in the test name, not only inside a comment that nobody sees when a boundary fails.

The failure usually starts with a helpful convenience method

Consider an agent that calls an API for issue data. The endpoint normally returns a small JSON page. The client sends Accept-Encoding: gzip, receives a response whose Content-Length is 18 KB, and logs the header as evidence that the result is modest. Its HTTP helper then transparently decompresses the body and calls ReadAll before JSON parsing.

A bad upstream response contains a large repeated description field in every record. The 18 KB transfer expands to far more data than the page would normally contain. The process allocates the byte slice, then creates strings and maps during parsing, then serializes selected records for the agent. Each stage holds a different copy for at least part of the request. A limit placed after parsing notices the problem only after the expensive work has happened.

The first repair often adds if len(body) > limit after ReadAll. That test makes the unit test look green while preserving the allocation spike. The second repair wraps the decoded reader, which is the correct direction, but still sends the first limit bytes to the agent as a fallback. Now the agent may take action based on a half response, and the audit trail does not show a clean failure.

The complete repair rejects early at the decoded reader, discards partial content, records the encoding and byte budget, and gives the caller an error it can classify. If the endpoint genuinely needs large exports, move that use case to an explicit download path with a higher approved limit, a destination chosen by the user, and no automatic insertion of the file into agent context.

That split matters because a report download and an agent tool lookup are different operations. Calling both an HTTP request does not make their risk or their budget the same.

Limits need an endpoint owner and a reason

A global default is a starting point, not a policy that fits every HTTP action. Assign a decoded response limit according to the endpoint's intended result. A status check may need only a few kilobytes. A paginated search can return a moderate structured response. A binary export may need a larger bound but should go to a file path rather than a conversational result.

Write the limit beside the action definition, along with the reason for it. "The API documentation says page size is 100" is not a sufficient reason because one record can still be enormous. A useful reason names the expected representation and the caller's use, such as "parse one status object and return selected fields" or "save a user requested archive after explicit approval."

Do not derive the cap from a server supplied page-size parameter alone. Servers can ignore a parameter, an agent can request a broad query, and a single text field can dominate the response. Cap page size, query breadth, decoded bytes, parsed records, and agent result bytes where each boundary makes sense. These controls overlap by design because each catches a different kind of bad request or bad response.

When you reject a response, record enough detail for diagnosis: HTTP method, host or action identifier, status code if available, declared content encoding, compressed bytes observed, decoded limit, and whether the decoder had begun producing output. Do not record the rejected body by default. Logging it can recreate the memory and sensitive-data issue in a supposedly diagnostic path.

A human approval does not make a response harmless. Approval may authorize an action with a remote service, but it does not predict how much data that service will return. Keep response bounds enforced after approval and before the agent receives any data.

Cancellation and timeouts cover a different failure mode

A decoded size cap stops growth after the decoder produces enough output. It does not stop a peer that sends bytes painfully slowly, a decoder that consumes excessive CPU on a crafted stream, or a response that never completes. Set request deadlines and make sure cancellation closes the response body and interrupts the reader.

Keep time, compressed bytes, decoded bytes, and parsed item counts separate in metrics and errors. If all failures appear as "request failed," someone will increase the size limit to solve a timeout, or raise a timeout to solve a parser failure. Those changes make incidents harder to diagnose and often expand the attack surface.

Test cancellation against an endpoint that sends a valid compressed prefix and then pauses. The client should return its timeout or cancellation error without keeping a goroutine waiting on the decoder. Then test a body that expands quickly past the decoded cap. That path should produce the size error promptly, even when the server would continue sending compressed bytes.

Connection reuse needs care after early rejection. In many clients, closing the body is enough to release resources, but the connection may not be reusable if the client has not consumed the rest of the response. That is acceptable. Correctness and bounded resource use beat squeezing one more keep-alive connection from a hostile or broken response.

Avoid retrying an oversized response automatically. A transient network error can justify a retry under a bounded policy. A response that exceeds a known limit is usually deterministic. Repeating it wastes bandwidth and can multiply the pressure on a process that is already dealing with an oversized action.

Parse limits belong after decode, not instead of it

A streaming JSON parser can stop you from storing every record, but it does not eliminate the decoded byte cap. Parsers still need buffers, individual strings can be huge, and error reporting can retain source fragments. Put the byte cap in front of the parser, then add format limits that match the data you accept.

For JSON, consider maximum nesting depth, maximum string length, maximum record count, and a strict schema for fields the agent will use. For CSV or line protocols, set a maximum line length and a maximum number of rows. For XML, disable external entity processing and set parser limits where the library provides them. These are parsing rules, while the decoded body cap is a transport boundary. Do not force one to impersonate the other.

Validate before rendering. A field named instructions, command, or message from a remote service is still remote content. Its size might be within the decoded cap and still be inappropriate to place in an agent's control flow. Select fields by schema and encode them as data. The size limit prevents one class of failure; it does not establish trust in what the bytes say.

This separation makes error handling less messy. A body that crosses the decoded limit should never reach the parser. A body that fits but has too many records should return a parsing or application limit error. A body that fits the schema but exceeds the agent presentation budget should be reduced by an explicit result rule. Each outcome tells an operator what needs changing, if anything.

Treat the audit record as a boundary report

A useful audit entry says what the process attempted and why the client refused to continue. It does not need the rejected content. Store the action identity, the authorization context, destination, request time, response status when known, encoding, wire byte count, decoded byte count or a lower bound, and the limit that ended the call.

For an early rejection, the decoded count may be limit + 1 rather than the actual final size because the client intentionally stopped reading. Record that honestly. Claiming to know the full decoded size suggests that you consumed the very stream you were meant to bound. A lower bound is enough to explain the decision.

This is particularly useful when an agent repeats an action after an operator changes a query. You can see that the first call exceeded the presentation or decoded budget and that the second one succeeded with a narrower request. Without that distinction, a failed tool call looks indistinguishable from an authentication or network problem, and people reach for the wrong fix.

Sallyport's activity trail can preserve the action result path without giving the agent the API credential itself. The caller still needs to report an oversized result as a bounded failure rather than turning the activity record into a second copy of the response.

Make the oversized case part of the release gate

Do not leave decompression checks as a security test that runs only when someone remembers it. Put a gzip and a Brotli boundary fixture in the normal client test suite. Run the same assertions through every HTTP code path that can return data to an agent, including redirects if your client follows them and error responses if it reads their bodies for diagnostics.

The assertions should be concrete. For a body under the limit, expect a complete decoded result and successful checksum validation. At the exact limit, expect the same. One byte over the limit, expect the typed size error, no parsed object, no agent result, and a closed response body. For malformed encoding, expect a decoding error. For an unsupported encoding, expect an explicit unsupported-encoding error before parsing begins.

Add a test for a compressed body with a low wire size and a decoded size over the cap. That is the test that catches the original mistake. A large uncompressed body only proves that a normal byte limiter works. It says nothing about the decoder boundary.

Review any change that introduces a new HTTP library, a new convenience fetch helper, or a new response logging path. Those changes routinely bypass the carefully limited reader because they look harmless in isolation. The code review question is simple: where do decoded bytes first become available, and what limit surrounds that exact reader?

If the answer is unclear, the implementation is not ready for an autonomous caller. A small response on the wire has earned no special trust. Make it prove its size after decoding, before it gets a chance to spend your memory or your agent's attention.