How we make a model
cheaper to run than to build.

Notes from the systems layer of Nexith — caching, memory, routing and privacy work that lets one model do more with less. We publish what we build, not benchmarks we can’t reproduce.

SystemsAug 2026

A semantic cache that makes repeat answers free

Most requests to an assistant are near-duplicates of earlier ones. We embed each prompt locally, store the vector in Postgres with pgvector, and serve a cached completion when cosine similarity clears a per-model threshold — the upstream is never called, so the marginal cost of that answer is zero.

Local embedding model — no per-lookup cost to a third party
Cosine recall gated at ≥ 0.92, bucketed by billing SKU
Cache hits save up to 90%: cached input billed at a fraction of the usual rate
Scoped to non-streaming single-turn calls where a hit is safe

SystemsAug 2026

Cross-session memory without leaking context

The gateway keeps two memory tracks: pinned facts written explicitly, and auto-extracted notes lifted from prior turns. Both are scoped to a user and application, embedded once, and retrieved by relevance — so a new conversation can start where the last one left off, without dragging the whole history back through the model.

Pinned vs. auto memory, each retrieved independently
Scoped by (user, application) — never cross-tenant
Inject / return / remember are opt-in per request
Retrieval threshold tuned per model against real traffic

InferenceAug 2026

Routing one public model across many backends

Behind a single public model name sits a pool of upstream providers. A lightweight classifier decides how much work a request needs, and the router picks a backend under fail-open policy — with per-key circuit isolation so one revoked credential cannot poison the pool for everyone else.

Per-key circuit breakers isolate a bad credential
Fail-open routing: a flaky backend degrades, never blocks
Rate limits and 429s tracked separately from hard failures
Hot catalog reloads preserve circuit and throttle state

PrivacyAug 2026

A single gate for what leaves the gateway

Every response path — streaming, non-streaming, cached replay, embeddings — passes through one allowlist scrubber before it reaches a client. Upstream provenance and internal telemetry are stripped at the wire, so the surface a caller sees is the surface we intend, on every exit.

One scrubber covers all exits, including cached replays
Streaming redaction holds back and re-checks partial frames
Nested telemetry fields stripped, not just top-level ones
Verified against a probe suite before each release

The best way to understand them is to build.

Each note above opens a deeper write-up on the system behind it. When you’re ready, the fastest way to see it work is to send a request.