A single gate for what leaves the gateway

The gateway calls many upstream providers, each with its own response schema. Some include diagnostic fields that name the model or the region. Some nest usage metadata several layers deep. If we forward those responses verbatim, we leak details about which backend served a request, how the router made its decision, and where the compute actually ran. Egress privacy is the system that prevents that.

One scrubber, every exit

There is a single function — scrub_upstream_response — that sits between the upstream and the client. It runs on every response, regardless of path: non-streaming chat, streaming chat, cached replay, embeddings. The scrubber is an allowlist, not a denylist. Only the fields we explicitly name are forwarded. Everything else is dropped.

• •Top-level fields like id, object, created, model are forwarded exactly.
• •Nested fields like choices[].message.content are recursively copied if they appear in the allowlist.
• •Usage metadata is flattened: only prompt_tokens, completion_tokens, and total_tokens. Any other usage fields — like prompt_tokens_details, cached_tokens, latency_checkpoint — are stripped.
• •System-specific metadata (provider_id, routing_tier, circuit_state) never appears in the allowlist. They exist only for internal logging and telemetry.

Streaming redaction holds back and re-checks

Streaming responses are harder. The gateway receives Server-Sent Events from the upstream, and each event is a JSON fragment: a delta of new tokens, a usage update, a [DONE] marker. We cannot wait until the stream finishes before scrubbing — that would destroy the latency benefit of streaming — so the scrubber must process each frame individually.

The solution is a hold-back buffer. Each frame is parsed, scrubbed, and held in memory. If the next frame is also valid, the previous frame is flushed to the client. If the next frame is [DONE] or an error, the buffer is drained and the connection closes. This approach ensures that a malformed frame — one that might contain unexpected fields — never reaches the client before we have confirmed it matches the expected schema.

Verified before every release

Egress privacy is not a feature you test once. Every new response path is a potential leak. We maintain a suite of probe requests that explicitly try to trigger each known upstream metadata field:

• •A chat request that hits a provider known to return latency_checkpoint in usage.
• •A streaming request that includes reasoning tokens in the delta.
• •A cached replay that originally included circuit state in the stored response.
• •An embeddings call to a provider that nests model_version inside metadata.

Each probe asserts that the response contains only allowlisted fields. If a field leaks, the test fails and the build stops. The suite runs in CI before every deployment, so a regression cannot reach production unnoticed.

The result is a system where we can confidently add new upstream providers, new response schemas, and new routing logic — and still guarantee that what the client sees is exactly what we intend them to see, on every exit.