Skip to main content
Görkem Tanağardıgil
← Writing

Local LLM · Frontier models · Evaluation

Local LLMs vs Frontier Models: Where Each One Wins

2026-01-28 · 8 min

When I run models on my own box, when I call Claude or GPT, and how the eval harness decides — not the hype cycle.

Frontier models (Claude, GPT-class APIs) are absurdly capable. Local models are absurdly private, controllable, and increasingly "good enough" for narrow loops.

The mistake is treating that as a tribe war. It is a routing problem.

TaskRouterLocal LLMFrontier APIEval gate
Fig — Route by data boundary, task shape, and shared eval gate

When local wins

  • Sensitive corpora — policies, payroll rules, customer tickets that should not leave the VPC
  • Tight loops — classification, routing, rewrite-for-retrieval, JSON extraction where latency and cost dominate
  • Offline / air-gapped evaluation runs against a frozen fixture set
  • Deterministic harnesses — same weights, same temperature, same seed where the stack allows

I use local models as specialists in the graph: intent routers, citation checkers, chunk taggers. They do not need to write poetry. They need to be stable.

When frontier wins

  • Ambiguous multi-step reasoning across messy docs
  • Tool-using agents that must recover from weird tool output
  • Writing and synthesis where taste and long-context matter
  • Bootstrapping a new domain before you have enough labeled failures to train or fine-tune anything

Anthropic-style agent writing keeps circling the same truth: the harness matters as much as the model. A weak local model in a strong graph with tools and evals can beat a frontier model in a sloppy prompt loop.

The decision checklist I actually use

  1. Does the data leave our boundary? If no → prefer local or VPC-hosted
  2. Is the task graded by exact structure (JSON schema, enum)? Prefer local + validator
  3. Is the task graded by groundedness against our KB? Either model, but shared eval
  4. Are we still discovering the task shape? Frontier first, then distill the pattern back to local

How we keep them honest

Both paths go through the same evaluation gate:

  • Schema validation
  • Groundedness / citation checks for RAG answers
  • Golden fixtures for regression

If a local model fails the gate, we do not "prompt harder" forever — we either improve retrieval, narrow the task, or escalate that node to a frontier call.

Bottom line

Local is for control and cost. Frontier is for judgment under ambiguity. The product is the router + eval, not the logo on the API key.