Agent Evals Before Demo Day: What We Measure Now

The first agent demo always looks magical. The second week in production looks like a crime scene.

We shipped a support-shaped agent that answered fluently and cited "sources." Users trusted it. Then a citation pointed at a chunk that did not contain the claim. Fluency had been grading itself.

What we borrowed from serious agent eval practice

Teams that live with agents in production (the Anthropic engineering notes on agent evals are a good public map) keep stressing the same shift: you are not grading a chatbot reply — you are grading a harness + model trajectory.

We adopted a simpler version of that mindset:

Outcome checks

• Did the agent resolve the intent or correctly escalate?
• For factual asks: exact or rubric match against a known answer key

Process checks

• Tool calls: right tools, reasonable count (no death spirals)
• Retrieval: did we fetch before claiming?
• Citations: claim ⊆ chunk

Trajectory review

• Store failing transcripts
• Tag root cause: retrieval / prompt / tool schema / model limit
• Promote the case into CI

LLM-as-judge — used carefully

A single judge prompt scoring groundedness and citation accuracy beat a committee of vague vibes. We keep the rubric short and tie it to retrievable evidence, not eloquence.

Where the answer is objective ("what is the freeze window for country X?"), the judge only checks equivalence to the key. Where the answer is open-ended, groundedness dominates.

LangGraph made failures visible

Before the graph, failures were "the model messed up." After:

• We saw retries stuck in one node
• We saw tools called with empty arguments
• We saw handoff nodes never reached

Evals attached to nodes, not just the final string. That is the difference between debugging a paragraph and debugging a system.

What we refuse to ship on

• No groundedness gate on RAG answers
• No conflict metric on concurrent claims (see the Mongo race post)
• No transcript store for the last N failures

Demo day can impress. Eval day keeps the pager quiet.