Routing one public model across many backends

Nexith exposes one model name — nexith-core, nexith-pro — but each is served by multiple upstream providers running different underlying weights. Some are hosted APIs, some are self-deployed inference servers. When a request arrives the gateway must decide which provider to call, and it must do so in a way that survives transient failures, respects rate limits, and isolates bad credentials from good ones.

A classifier picks the difficulty tier

Not every request needs the most expensive model. Simple questions can be routed to a cheaper, faster backend. The L1 classifier is a small local model — currently Qwen 2.5 3B, chosen because it never hallucinates about task difficulty — that reads the prompt and returns a routing decision: direct (simple, factual) or reasoning (complex, multi-step).

The classifier itself costs almost nothing to run. Inference takes about 0.5 seconds, and because it is local there is no per-request billing. Misclassifications are bounded by policy: if the classifier says "direct" but the request would benefit from reasoning, the answer might be less thorough than ideal — but it will still be coherent, and the caller paid a fraction of what a reasoning call would cost.

Per-key circuit breakers isolate failures

A provider can fail in several ways: the API key is revoked, the endpoint is down, rate limits are hit. The naive approach is a global circuit breaker that opens after N failures and stops routing requests to that provider. The problem with that approach is that it treats all keys the same. If one customer's key is revoked, every other key in the pool is effectively penalised — because successful calls from good keys reset the shared failure counter, and the breaker never trips.

We use per-key circuit breakers instead. Each (provider, api_key) pair tracks its own failure count, independently. A revoked key will hit the failure threshold and enter quarantine — at which point the router stops selecting it — but good keys continue to route normally. The pool remains healthy even when one credential is bad.

• •401 Unauthorized is treated as Unavailable, not a hard failure. The key might be revoked, or the request might be malformed. Either way, the router fails over to the next provider.
• •429 rate-limit responses are tracked separately from hard failures. A provider that is merely throttled should not be permanently disabled — it should back off and retry later.
• •Transient failures (timeouts, 5xx errors) are counted, but not indefinitely. The breaker uses an exponentially weighted moving average to decay old failures, so a provider that recovers is automatically re-enabled.

Hot catalog reloads preserve circuit state

The gateway reloads its routing catalog every 5 seconds, so operators can add a provider, update a model mapping, or change a rate limit without restarting the service. The reload builds a new provider pool from scratch — which would normally reset all circuit breakers and throttle state, allowing a bad provider to immediately pollute the pool again.

The solution is state inheritance. When the new pool is built, the gateway walks the old pool and copies circuit state, throttle state, and per-key health from any (public_model, provider_id, provider_model) tuple that exists in both. The only exception is half-open probes — requests that are in-flight testing whether a previously broken circuit has recovered — which are never inherited, because copying "true" would freeze the breaker in the half-open state forever.

The result is a router that survives configuration changes, credential failures, and transient errors without dropping requests or letting one bad key poison the pool.