RAG That Earns Citations: What Broke Before Agents Got Useful

Everyone can wire an embedding API to a vector store by Friday. The hard part is the Monday after: a support agent cites the wrong policy version, a chat bot invents a refund rule, and nobody can say whether the model lied or retrieval handed it garbage.

This is the build log for a knowledge base meant to feed specialized agents — support, in-product chat, domain workflows — not a demo chatbot.

The problem we actually had

Institutional knowledge lived in PDFs, Confluence exports, ticket macros, and tribal Slack threads. A generic frontier model could sound confident. It could not be accountable.

We needed three properties:

1. Grounded — every claim should map to a retrieved chunk
2. Fresh — policy v3 must beat policy v2 in retrieval
3. Specializable — the same spine should power support vs payroll-domain agents

Challenge 1 — Chunking that respects meaning

Naive fixed-size chunks split tables mid-row and severed "if / then" policy clauses. The model then answered from half a rule.

What we changed:

• Structure-aware splits (headings, list items, table rows) before token windows
• Overlap only across section boundaries, not arbitrary mid-sentence cuts
• Metadata on every chunk: doc_id, version, effective_from, audience (support | internal | legal)

Version metadata later became the difference between citing the live refund policy and a retired draft.

Challenge 2 — Semantic search alone was not enough

Embeddings are great at "things that feel related." They are mediocre at exact tokens: SKUs, article IDs, error codes, regulation names.

Hybrid retrieval:

• Dense vector search for semantic neighbors
• Sparse / keyword path for identifiers and titles
• Reciprocal rank fusion to merge lists
• A hard filter on audience and effective_from <= now

Without the filter, stale docs ranked high because they were longer and phrase-heavier.

Challenge 3 — "It sounds right" is not an eval

We stopped eyeballing answers. Inspired by how serious agent teams grade outputs (factuality, citations, coverage), we built a thin harness:

• Groundedness: does each sentence have a supporting chunk?
• Citation accuracy: does the cited chunk actually contain the claim?
• Coverage: did we miss a required policy clause for this intent?

Failing cases went into a regression set. Prompt tweaks that raised fluency but dropped groundedness were rejected.

Challenge 4 — One corpus, many agents

Support agents need runbooks and escalation paths. Domain agents need schemas and calculation notes. Same RAG spine, different:

• Tool allowlists
• System prompts
• Exit conditions (answer / ask clarifying / hand off to human)

LangGraph made the branching explicit. Retrieval stayed shared; specialization lived in the graph.

What I would not skip next time

• Versioned metadata from day one
• Hybrid retrieval before fancy re-rankers
• A groundedness eval before any "launch" demo
• Human handoff as a first-class graph node, not an apology path

RAG is not a feature you toggle. It is an operations practice: ingest, retrieve, cite, measure, repair.