Everyone demoing a retrieval-augmented chatbot makes it look easy. Getting one to hold up in production is a different job — and almost none of it is the part people find exciting.
The prototype that wowed the room usually breaks the moment real users arrive with real questions. The failure is rarely the model. It's the plumbing around it: how documents are split, how retrieval is scored, and whether anyone is measuring quality at all.
Chunking is a product decision
How you split source documents decides what the model can ever retrieve. Chunk too small and you strip away the context a passage needs to make sense; too large and you dilute the signal with noise. There is no universal size — it depends on your documents and the questions people actually ask.
We've seen teams copy a chunk size from a blog post and wonder why retrieval feels random. A support-ticket knowledge base and a legal contract corpus don't chunk the same way, because the unit of meaning in each is a different length — treat chunking as a design decision specific to your content, not a default to inherit.
Retrieval quality decays quietly
A RAG system that scored well at launch doesn't stay that way for free. Source documents get updated, edge-case questions accumulate, and embeddings drift out of sync with how users actually phrase things six months later. Without a way to notice, the first sign of decay is a customer complaint, not a dashboard.
If you can't measure retrieval quality, you're not tuning a system — you're guessing with extra steps.
Evals before vibes
The single biggest predictor of whether a RAG system survives contact with users is whether the team built an evaluation set early. A few hundred real questions with known-good answers turns every future change from a leap of faith into a measured decision.
We treat evals as a first-class deliverable, not an afterthought. Every change to chunking, embeddings, or prompts runs against the set before it ships, so quality moves in one direction.
Monitoring in production
Once it's live, the eval set stops being enough on its own — real user queries will find gaps no offline test anticipated. We log retrieval scores and flag low-confidence answers for review, so degradation shows up as a metric trending the wrong way, not as a support ticket days later.
Reranking is cheaper than a bigger model
When retrieval quality plateaus, the instinct is to reach for a larger embedding model or a bigger context window. Often the actual gap is between what's retrieved and what's ranked highest — a reranking pass over the top candidates fixes more relevance problems than a more expensive first-stage retriever, for a fraction of the cost.
It's an unglamorous middle step, sitting between retrieval and generation, and it's easy to skip because nothing about it demos well. But it's usually the highest-leverage half-day of work available once the obvious chunking and embedding choices are already in place.
Users will ask questions your documents don't answer
No knowledge base covers everything a real user eventually asks, and a RAG system's failure mode for that gap matters more than its average-case performance. A system that confidently answers from nothing looks worse to a user than one that says it doesn't know — the second failure is recoverable, the first erodes trust in every correct answer that came before it.
We treat "I don't have enough information to answer that" as a valid, designed-for output, not an edge case to eliminate. Retrieval confidence scores below a set threshold route there by default, rather than letting the model paper over a weak retrieval with a fluent-sounding guess.
We'll pressure-test your retrieval quality before your users do.
Ship the unglamorous work
Caching, monitoring, fallback behavior when retrieval comes back empty — none of it demos well, but all of it is the difference between a toy and a tool. Get the boring parts right and the impressive parts take care of themselves.
