Cross-session memory without leaking context
The gateway remembers what matters from earlier conversations, scoped to a user and application, so a new session can start where the last one left off — without dragging the entire history through the model every time.
A stateless model has no memory. Every conversation is a cold start. Users expect assistants to remember their preferences, their codebase, their last question — but sending the full conversation history into every request becomes prohibitively expensive once those histories span dozens of turns. Session memory is the second savings layer, and it solves that: persist the facts that matter, retrieve them by relevance, and inject only what the current turn actually needs.
Two memory tracks
The gateway maintains two kinds of memory, each with a different write path and a different role:
- •Pinned memory: facts written explicitly by a user or an API call. Never expires, never extracted automatically. Always injected when present.
- •Auto memory: notes extracted from prior turns by a local model that watches for statements worth persisting. Embedded once, retrieved by cosine similarity when relevant.
Both are scoped to a (user_id, application_id) pair, so one tenant never sees another tenant's memories, and one application never contaminates another. Retrieval happens independently — the gateway might inject two pinned facts and three auto-extracted notes if those five clear the relevance threshold.
Opt-in, not automatic
Memory injection is controlled by three request-level flags: inject, return, and remember. All three default to false. This is deliberate. An API caller might want to retrieve memories without writing new ones, or write a memory without polluting the immediate response, or skip memory entirely for a one-off disposable question.
Injecting memory into a prompt voids eligibility for the semantic cache. The effective input is no longer the visible text — it is the text plus the hidden context — so a cache hit could serve an answer that assumed different memories.
Extraction runs in the background
After a turn completes, the gateway decides whether to extract a memory from it. Extraction runs locally — never sent to a paid third party — with a 30-second budget. If the model exceeds that, extraction is silently skipped. The process is entirely invisible to the caller; no latency is added to the request path.
A small embedding model (same one used for cache lookups) encodes each extracted memory as a 768-dimension vector. When a later request asks to inject memories, the gateway embeds the new prompt, retrieves the nearest auto-memories by cosine similarity, and includes any that clear a per-model relevance threshold — typically 0.40, but tuned against real traffic for each upstream.
Where the third token goes
Injected memories are prepended to the prompt before it reaches the model. The caller is billed for those tokens, because the model is processing them — but the input cost is offset by the fact that we are not resending the full conversation. A three-turn exchange without memory might burn 4,000 tokens of repeated history. With memory, that becomes 300 tokens of retrieved facts, injected only when relevant.
The result is a system that lets assistants feel stateful without actually being stateful, and that does it at a token cost lower than the naive alternative of context-stuffing every prior turn.