RAG in a demo vs. RAG in production

Retrieval augmented generation, usually shortened to RAG, is the pattern behind most useful AI applications that work with your own data instead of the model's general knowledge. It is also one of the most deceptive patterns there is. A RAG demo is trivially easy to build. A RAG system that actually works in production is genuinely hard. The distance between those two is where a great many AI projects quietly die, after an exciting demo and a long, confusing struggle to make the real thing reliable.

What RAG is, briefly

The idea is simple. Instead of asking a model to answer from its own general knowledge, you first retrieve the relevant pieces of your own information. Then you hand those pieces to the model along with the question. Its answer is now grounded in your actual data. That is what lets an AI assistant answer questions about your specific documents, your product, or your business, rather than making things up from general training. The concept is easy to grasp, which is part of why the difficulty of doing it well catches people off guard.

Why the demo is easy

Building a RAG demo takes an afternoon. You take a handful of clean documents. You set up basic retrieval. You ask a few questions you already know the documents can answer, and it works beautifully. Everyone is impressed.

It works so easily because a demo quietly controls all the hard variables. The data is small and clean. The questions are ones you chose because they have clear answers in the material. The volume is one. Under those conditions RAG looks close to solved. That impression is the trap.

What breaks in production

Take the same approach into production, with real data and real users, and it comes apart in several places at once:

• Retrieval quality at scale
• Messy real data
• How you split the content
• Keeping the information fresh
• Evaluation
• The questions users actually ask

Retrieval quality at scale is the big one. When you have not a handful of documents but a large and growing collection, finding the genuinely relevant pieces for a given question becomes the whole problem. If retrieval pulls the wrong material, the model gives a confident answer grounded in the wrong information. That is worse than no answer at all. The quality of retrieval is the ceiling on the quality of everything else.

Real data is messy. It is inconsistent, duplicated, outdated, poorly formatted, and full of contradictions. All of that mess flows straight into retrieval quality, and cleaning it up is a large part of the actual work the demo never had to face.

How you split the content matters more than it should. Long documents have to be broken into pieces to be retrieved, and how you split them decides whether the right piece can be found at all. Split poorly and the relevant answer is scattered across fragments or buried in noise. It is an unglamorous decision with an outsized effect on whether the system works.

Freshness is easy to forget. In a demo the data is static. In production it changes, and the retrieval system has to stay current as it does, or it starts confidently serving answers from information that is no longer true.

Evaluation is what tells you the truth. As with any AI in production, you cannot know whether retrieval and answers are holding up at scale unless you build a way to measure it. Without that, quality degrades silently until users lose trust.

Then there are the questions users actually ask. They ask things you did not anticipate, phrased in ways you did not plan for, including many the underlying data cannot answer. Handling the full messy range of real questions is a different problem from handling the few you picked for the demo.

The retrieval is the product

The single most important thing to understand about production RAG is that the retrieval, not the model, is where the real work lives. Everyone focuses on the model because it is the exciting part. But the model is largely a fixed ingredient you plug in. The part you actually build, tune, and sweat over is getting the right information in front of the model at the right time.

A brilliant model fed the wrong context produces confident nonsense. A modest model fed exactly the right context produces something genuinely useful. In production RAG, retrieval quality is the product, and treating it as an afterthought is why so many of these systems disappoint.

What I learned building it

I built a RAG system as the backend of an internal tool for matching client requirements against a large network of candidates. It read the requirements and surfaced the strongest matches. The lesson was exactly this. The model was the easy part. The hard, valuable work was in the retrieval, the data quality, and everything around getting the right information into the model's hands reliably and at scale.

The same held true building AI agents into Custom1, a tool I built for designing custom protective foam. The agents were only as good as the information and structure feeding them. In both cases the demo would have been quick, and the production system was the real project.

If you are planning to build on RAG, plan for the production version, not the demo. Budget your effort for retrieval quality, messy data, freshness, and evaluation, because those are the parts that decide whether you have a real product or an impressive afternoon that never quite becomes trustworthy. The demo will lie to you about how close you are. The retrieval will tell you the truth.