Production RAG From Scratch: A Consultant's Deep Dive

It is Saturday night and you are staring at a job req that says 'experience building production RAG systems required.' You have played with LangChain, you have wired up a Chroma index against a PDF, and it mostly worked. That is not what this req is asking about.

Production RAG is a different animal. It has to survive messy source documents, adversarial user queries, latency budgets, and someone in procurement asking why the bot hallucinated a contract clause. This deep dive walks through the five decisions that separate a demo from something a client will actually put in front of employees or customers.

No fluff, no toy notebook. Just the architecture choices you will be asked about in a technical screen, and the reasoning behind each one.

1. Chunking Strategy: Where Most RAG Systems Quietly Fail

Chunking is the decision nobody wants to spend time on and the one that breaks retrieval quality most often. Fixed 512-token windows with no overlap look fine in a demo and fall apart the moment a table, a numbered clause, or a code block gets split mid-structure.

• Recursive character/token splitting (LangChain's RecursiveCharacterTextSplitter or similar) with 200-400 token chunks and 10-15% overlap is a reasonable default for prose-heavy documents.
• Semantic chunking — splitting on embedding similarity drops between sentences — performs better on technical manuals and policy documents where topic boundaries matter more than paragraph breaks.
• Structure-aware chunking is non-negotiable for PDFs with tables, contracts with numbered sections, or code repos. Use a layout-aware parser (Unstructured, PyMuPDF, or Azure Document Intelligence) before you chunk, not after.
• Metadata at chunk level — source document, page number, section heading, effective date — is what lets you filter retrieval and cite sources credibly. Skip this and your answers become unauditable.

Rule of thumb: if a human SME reading a chunk in isolation cannot tell what it is about, your retriever cannot either.

2. Choosing an Embedding Model

The embedding model decides what 'similar' means in your vector space. Get it wrong and no amount of reranking will save you.

For a client with strict data residency requirements (health care, defense, financial services), a self-hosted model like bge-large or a fine-tuned variant is often the only viable path, even if it costs more engineering time upfront. For everything else, start with a hosted API model and measure before you optimize.

3. Hybrid Retrieval: BM25 Plus Dense Vectors

Pure dense retrieval loses on exact-match queries — product SKUs, error codes, legal citation numbers, part numbers. Pure keyword search (BM25) loses on paraphrase and conceptual queries. Production systems use both and fuse the results.

1. Run the query through BM25 (Elasticsearch, OpenSearch, or a library like rank_bm25) to get a keyword-relevance ranked list.
2. Run the same query through your dense embedding index (pgvector, Pinecone, Weaviate, or Qdrant) for semantic similarity.
3. Fuse the two ranked lists — Reciprocal Rank Fusion (RRF) is the standard, simple approach and needs no tuning of weight parameters.
4. Pass the top 20-50 fused candidates forward to reranking.

If you only build one retrieval path, you are shipping a system that will fail predictably in a client demo the first time someone types an exact error code.

4. Reranking: The Step Everyone Skips

Retrieval gets you a candidate set. Reranking decides what actually goes into the prompt, and it is the highest-leverage, lowest-effort quality improvement available.

A cross-encoder reranker (Cohere Rerank 3, or an open model like ms-marco-MiniLM-L-6-v2 self-hosted) scores each query-passage pair jointly, rather than comparing pre-computed embeddings. This catches relevance signals dense retrieval misses, at the cost of extra latency — typically 50-150ms for 20-50 candidates, which is acceptable for most enterprise chat use cases.

Take the top 3-8 reranked passages, not the top 20. Stuffing the context window with marginally relevant chunks increases hallucination risk and cost without improving answer quality.

5. The Evaluation Harness

Without an evaluation harness you are guessing, and guessing does not survive a client's UAT phase. Build this before you tune anything else.

• Retrieval metrics: recall@k and MRR against a labeled set of query-to-ground-truth-chunk pairs. Fifty to one hundred hand-labeled queries per document domain is a realistic starting point for a consulting engagement.
• Generation metrics: faithfulness (does the answer stay grounded in retrieved context) and answer relevance, scored with a framework like RAGAS or a custom LLM-as-judge rubric.
• Regression testing: rerun the full eval set on every change to chunking, embedding model, or prompt template. This is the difference between an engineering practice and a lucky demo.

Clients will ask 'how do you know it works' before they ask 'how does it work.' Have the numbers ready.

Guardrails for Production

A RAG system without guardrails is a liability, not a product. At minimum, implement: input validation to reject prompt injection attempts embedded in retrieved documents, output filtering for PII leakage, a confidence threshold that triggers 'I don't have enough information' instead of a confident hallucination, and full citation of source documents so a human can verify every answer.

These are not nice-to-haves. They are usually the line item that gets a RAG pilot approved for production rollout or sent back for another quarter of hardening.

If you are weighing which of these skills to deepen first, or want to talk through how RAG experience is showing up in current contract requirements, the Josh Pros LLC team is happy to talk shop. Reach out at contact@joshpros.com or visit https://joshpros.com.

#RAG #ProductionRAG #RAGArchitecture #EnterpriseRAG #VectorSearch #LLMOps #AIEngineering #TechConsulting #ContractConsultants #MachineLearning #DataEngineering #SemanticSearch

Back to all insights