FIG. 5 — A local-first LLM router with a judge-gated escalation path
CoreThread
A self-hosted, OpenAI-API-compatible router that answers every request with a cheap local model first, has a second model grade that answer against a rubric, and calls a paid frontier model only when the local answer measurably fails.
- Status
- Working prototype · 255 commits
- Context
- 2026 · Independent work
- Role
- Sole designer & engineer
- Code
- Private — on request
A live trace: this request cleared the local model and the judge without touching a frontier API. Below, a deep route broken into planned steps with per-step timing — every routing decision visible and auditable.
The problem
Anyone running local models alongside frontier APIs faces a choice that shouldn't be binary: always cheap and local, or always pay for the best. The right answer changes per request, not per deployment.
The usual workaround is to guess — route by task type, or by some heuristic about difficulty. Guessing under-uses the local model on work it would have handled fine and over-trusts it on work it won't.
In an industrial or regulated setting there's a second cost to defaulting outward: every request that leaves by default is a data-locality decision nobody actually made.
How it works
- 01
Drop-in OpenAI-compatible
A FastAPI service exposing OpenAI-compatible chat-completions and models endpoints, streaming and non-streaming. Any tool already built against the OpenAI client points at it and works unchanged. The routing is invisible to the caller; if it weren't, people would route around it.
- 02
The judge answers yes-or-no questions
A second model grades the local answer against three boolean rubric items — did it answer the core question, is it free of hedging disclaimers, is it internally consistent — and those map to a deterministic score. The judge's own confidence float is ignored entirely: models are badly calibrated about their overall certainty, but they're reasonably good at answering a specific yes-or-no question about a specific piece of text.
- 03
At most one judge call and one frontier call
The orchestrator is written so that exceeding one judge call and one frontier call per request is structurally impossible, with a runtime assertion on every exit path as a backstop. I didn't want cost control to depend on anyone remembering to check a counter.
- 04
Defensive parsing and a fail-safe verdict
The judge's JSON goes through fence-stripping, brace-balancing, and one retry, then falls through to a sentinel verdict if it still won't parse. A grader that throws on malformed output turns a quality gate into an outage, so this one can't throw.
- 05
Prompt-injection defense at the grading step
The judge prompt explicitly instructs the grading model to ignore any instructions embedded in the question or answer it's grading. Both of those fields can carry attacker-controlled text, and a grader shouldn't take direction from the material it's evaluating.
- 06
Adapters and observability
Ollama, LM Studio, OpenAI, and OpenRouter sit behind a common interface, with model roles configured in YAML instead of hardcoded. A React SPA shows live traces, usage, and configuration, with opt-in rate limits, quotas, cost estimates, and audit logs. Traces carry no request bodies, and the global exception handler is written so bearer tokens can't leak through generic error messages.
General note
Process close to the source, keep data local by default, and escalate on defined exception criteria. That isn't an AI idea — it's how you design a well-behaved OT system, applied to inference.
Why it matters
Frontier spend attaches only to requests that demonstrably needed it, and the orchestrator's structure makes overspending impossible.
Data stays local by default, which is a hard precondition in regulated and industrial environments.
The escalation criteria are explicit and auditable. You can say exactly why any given request went where it went.
Body-free traces, exception handling that can't leak tokens, and an injection-resistant judge are what would let this sit on a real network.
Stack
- Service
- Python 3.12FastAPIPydantic v2uvicornhttpx
- Models
- OllamaLM StudioOpenAIOpenRouterLLM-as-judge
- Interface
- ReactViteServer-sent eventsstructlog
- Quality
- pytestmypyruffuv
Want the architecture in more depth, or a walk through the code? I'm glad to go there.
Get in touch