A semantic cache that makes repeat answers nearly free
Most requests to an assistant are near-duplicates of ones we have already answered. Here is how we recognise them, serve the earlier answer, and skip the upstream call entirely.
A production assistant sees the same question over and over — reworded, re-punctuated, but semantically identical. Answering each one from scratch means paying an upstream provider for compute we have already spent. The semantic cache is the first of three savings layers in the gateway, and it targets exactly this: turn a repeat request into a lookup instead of a generation.
Embedding, not string matching
A literal cache keyed on the prompt string catches almost nothing — "what is a vector database" and "explain vector databases" hash differently. So instead of hashing the text, we embed it. Every eligible prompt is run through a local embedding model that produces a 768-dimension vector, and that vector is what we store and compare against.
The embedding step runs on our own hardware. A cache lookup never sends anything to a paid third-party embedding API — otherwise the lookup itself would cost money and defeat the point.
Vectors live in Postgres with pgvector, indexed for approximate nearest-neighbour search. When a request arrives, we embed it, ask Postgres for the closest stored vector, and read back the cosine similarity of that match.
A threshold you can defend
Recall is gated at a cosine similarity of 0.92 or higher. Below that line we treat the request as new and generate a fresh answer. The threshold is deliberately conservative: a false hit — serving the wrong cached answer — is far more damaging than a miss, which merely costs a normal generation.
- •Candidates are bucketed by billing SKU, so a cheap model never serves an answer generated by a premium one.
- •Only non-streaming, single-turn calls are eligible — the cases where a cached reply is unambiguously safe.
- •Injecting session memory into a prompt voids cache eligibility, because the effective input is no longer the visible text.
What a hit actually costs
This is where we are careful with language. A cache hit is not free, and we never advertise it as "$0". On a hit the upstream is never called, so there is no generation cost — but the request is still billed the way OpenAI bills a cached input: a fraction of the usual input rate, plus the output. In practice that lands at up to 90% off the equivalent uncached call.
Marketing says "save up to 90%," never "free." The billing path and the copy agree on purpose — a claim the invoice contradicts is worse than no claim.
The result is a layer that quietly absorbs the most repetitive traffic on the platform, at a cost that is honest about what a lookup-plus-billing actually is.