You built a retrieval-augmented generation demo. It answered questions from a folder of PDFs, the client nodded, and everyone called it a win. Then the client asked for it on 400,000 documents, sub-two-second responses, and a way to prove it does not make things up. That is where most demo builders stall.
Clients hiring for RAG work today are not impressed by a LangChain notebook. They have seen a dozen of those. What they are screening for is whether you have shipped the unglamorous parts: retrieval that actually retrieves the right thing, guardrails that hold under adversarial input, and cost math that does not blow the budget in month two.
This is a walkthrough of what separates a production rag system from a proof of concept, and the questions clients use to tell the difference in an interview.
Chunking and Metadata Strategy: The Part Clients Pay to Get Right
Fixed-size chunking with a little overlap works fine on twenty clean documents. It falls apart on two hundred thousand documents with mixed formats, revision histories, and access-control requirements.
Production chunking decisions that actually get discussed in scoping calls:
- Semantic chunking versus fixed-token chunking, and when each one produces better recall for the domain
- How tables, code blocks, and headers are preserved instead of shredded mid-structure
- Overlap sizing so answers do not get cut across a chunk boundary
- Metadata tagging at ingestion time, not bolted on later
Metadata is the part demo builders skip entirely, and it is the part that lets a retrieval system filter before it searches. At minimum, production pipelines tag each chunk with source document, publish or revision date, document owner or department, and access tier. Without that last field, you cannot build a system that respects who is allowed to see what, which is a hard requirement the moment a client has more than one customer or one internal audience.
Retrieval Evaluation: Proving Recall Instead of Eyeballing It
Demo builders test retrieval by asking a few questions and reading the answers. Production teams build a golden question set with known correct source chunks, then measure precision and recall at k against it every time the pipeline changes.
Open-source evaluation frameworks such as RAGAS and TruLens, along with vendor-provided eval tooling, give you a repeatable way to score faithfulness, answer relevance, and context precision without a human reading every output by hand. The specific tool matters less than whether you can answer this in an interview: how do you know your retrieval got worse after you changed the embedding model?
If the honest answer is you do not know, that is a signal you have not run a production retrieval pipeline. Teams that have will describe a regression suite, a dashboard, and a threshold that blocks a deploy.
Hallucination Guardrails Clients Actually Ask About
Every client asks some version of how do you stop it from making things up. The demo-only answer is a system prompt that says only answer from the provided context. That helps, but it is not a guardrail on its own.
Production guardrails layered in real systems typically include:
- Citation grounding, where the model must reference the specific chunk it drew from, and the UI surfaces that source so a human can verify it
- A confidence or relevance threshold on retrieved chunks, below which the system returns an explicit I do not know rather than a low-quality answer
- A refusal or escalation path for questions outside the indexed corpus
- Human review loops for high-stakes domains like healthcare, legal, or financial guidance, where an ungrounded answer carries real liability
Clients in regulated industries will ask specifically how ungrounded answers are caught before they reach a user. If your answer stops at prompt wording, you are describing a demo, not a control.
Latency and Token Cost Budgets: The Math That Kills Demos
A demo answering one question at a time with no load has no cost or latency pressure. Production systems carry both, and clients want to see that you planned for it before launch, not after the first invoice.
| Pipeline stage | What drives the cost or delay | What production teams tune |
|---|---|---|
| Embedding at query time | Model size, network round trip | Caching common queries, smaller embedding models for retrieval-only tasks |
| Vector search | Index size, filter complexity | Metadata pre-filtering to shrink the search space before the vector call |
| Reranking | Number of candidates passed to the reranker | Limiting candidates to a top-N set rather than reranking everything retrieved |
| Generation | Context window size, output length | Trimming injected context to only the chunks that scored above threshold |
Every extra chunk stuffed into the prompt is tokens you pay for on every single query, multiplied by every user, every day. Clients that have run this in production will ask you directly what your average tokens per query looked like and how you brought it down. Vague answers here are the clearest tell of demo-only experience, because cost only becomes a real constraint at scale.
Observability: What Breaks After Launch
A demo does not need monitoring because someone is watching it live. A production system runs unattended, and things drift quietly. Source documents get updated and the index goes stale. A model provider ships a silent update and answer quality shifts. Retrieval starts returning the wrong chunks for a query pattern nobody tested.
What production observability actually looks like:
- Logging every retrieved chunk alongside the final answer, so a bad response can be traced back to a retrieval failure or a generation failure
- Tracing latency by pipeline stage, not just end to end
- Automated re-embedding triggers when source documents change, instead of a manual reindex someone forgets to run
- Alerting on retrieval score drops or a spike in I do not know responses, both of which usually mean the corpus or the query patterns have shifted
If your only monitoring plan is checking the app periodically, that is a demo mindset carried into a production conversation.
Three Questions Clients Use to Spot Demo-Only Experience
Technical screeners in this space have converged on a short set of questions because the answers are hard to fake without having actually run one of these systems.
- How did you measure whether your retrieval improved after a change? A real answer names a metric, a golden set, and a before-and-after comparison. A demo answer describes a vibe check.
- What happened when a source document was updated or deleted? A real answer describes a re-ingestion or invalidation process. A demo answer has never encountered the question because the demo corpus never changed.
- What was your average latency and token cost per query, and what did you do about it? A real answer has numbers, even rough ones, and a story about a tradeoff made. A demo answer has neither, because cost was never a constraint on a weekend project.
None of these questions are tricks. They are simply things you cannot answer without having run a retrieval system past the prototype stage, under real load, with real documents that change.
If you are weighing whether to invest time in RAG evaluation tooling, vector database internals, or cost-optimization work before your next contract search, the team at Josh Pros LLC talks with consultants and clients on both sides of these conversations every week. Reach out at contact@joshpros.com or visit https://joshpros.com if you want a candid read on what is actually being asked for right now.
#RAG #RetrievalAugmentedGeneration #ProductionRAG #AIEngineering #LLMOps #VectorSearch #MLOps #TechContracting #ITConsulting #AICertifications #RAGInterview #DataEngineering
Talk to a real recruiter, not a bot.
We'll tell you the rate, the client, and the terms before you interview. And if we're not the right fit, we'll say so.