Skip to content
All work

A local-first LLM router with a judge-gated escalation path

CoreThread

A self-hosted, OpenAI-API-compatible router that answers every request with a cheap local model first, has a second model grade that answer against a rubric, and calls a paid frontier model only when the local answer measurably fails.

Status
Working prototype · 255 commits
Context
2026 · Independent work
Role
Sole designer & engineer
Code
Private — available on request
01The problem

Anyone running local models alongside frontier APIs faces a binary that shouldn't be binary: always cheap and local, or always pay for the best. The trade-off is real per request, not per deployment.

The usual workaround is to guess — route by task type, or by some heuristic about difficulty. Guessing under-uses the local model on work it would have handled fine, and over-trusts it on work it will not.

In an industrial or regulated setting there is a second cost to defaulting outward: every request that leaves by default is a data-locality decision made by omission rather than on purpose.

02How it works
  1. 01

    Drop-in compatible on purpose

    A FastAPI service exposing OpenAI-compatible chat-completions and models endpoints, streaming and non-streaming. Any tool already built against the OpenAI client points at it and works unchanged — the routing is invisible to the caller, which is the only way a router like this gets adopted rather than worked around.

  2. 02

    A judge with a rubric, not a confidence score

    A second model grades the local answer against three boolean rubric items — did it answer the core question, is it free of hedging disclaimers, is it internally consistent — which map to a deterministic score. The system ignores the judge's own confidence float entirely. Models are poorly calibrated about their own certainty; they are far better at answering a specific yes-or-no question about a specific piece of text.

  3. 03

    An invariant enforced by code shape

    The orchestrator is written so that exceeding one judge call and one frontier call per request is structurally impossible, backed by a runtime assertion on every exit path. Cost control that depends on remembering to check a counter eventually fails; cost control the shape of the code will not permit does not.

  4. 04

    Defensive parsing and a fail-safe verdict

    The judge's JSON goes through fence-stripping, brace-balancing, and one retry, then falls through to a sentinel verdict if it still will not parse. A grader that throws on malformed output turns a quality gate into an outage.

  5. 05

    Prompt-injection defense at the grading step

    The judge prompt explicitly instructs the grading model to ignore any instructions embedded in the question or answer it is grading. A judge that reads attacker-controlled text and treats it as direction is a gate that opens itself.

  6. 06

    Adapters and observability

    Ollama, LM Studio, OpenAI, and OpenRouter sit behind a common interface, with model roles configured in YAML rather than hardcoded. A React SPA shows live traces, usage, and configuration, with opt-in rate limits, quotas, cost estimates, and audit logs. Traces carry no request bodies, and the global exception handler is built so bearer tokens cannot leak through generic error messages.

Design principle

“Process close to the source, keep data local by default, and escalate on defined exception criteria. That is not an AI idea — it is how you design a well-behaved OT system, applied to inference.”
03Why it matters

  • Inference cost is contained by construction rather than by discipline — frontier spend attaches only to requests that demonstrably needed it.

  • Data stays local by default, which is the posture regulated and industrial environments require before any of this is deployable.

  • The escalation criteria are explicit and auditable. You can say why any given request went where it went, which matters the moment someone asks.

  • The security details — body-free traces, non-leaking exception handling, an injection-hardened judge — are the difference between a demo and something that could sit in a real network.

04Stack
Service
Python 3.12FastAPIPydantic v2uvicornhttpx
Models
OllamaLM StudioOpenAIOpenRouterLLM-as-judge
Interface
ReactViteServer-sent eventsstructlog
Quality
pytestmypyruffuv

Next project

Deterministic P&ID Extraction

Want the architecture in more depth, or a walk through the code? I'm glad to go there.

Get in touch