RAG Implementation Checklist

1

Document Processing

Audit your document corpus

Inventory all document types (PDF, HTML, DOCX, Markdown, CSV), their volume, and update frequency. Identify which documents are most critical for user queries.

Why it matters: Different document types need different parsers. A corpus audit prevents building a pipeline that handles 80% of documents well but completely fails on the 20% users care about most.

Common pitfall: Assuming all documents are clean text. Scanned PDFs, tables, images with text, and multi-column layouts all require specialized extraction.

Implement robust document parsing

Use specialized parsers for each document type. For PDFs, use libraries that handle OCR, tables, and layout detection (e.g., Unstructured, PyMuPDF). For HTML, strip boilerplate and extract main content.

Why it matters: Garbage in, garbage out. If your parser mangles tables into random text or loses section headers, no amount of embedding sophistication will fix retrieval quality.

Common pitfall: Using a simple PDF-to-text converter that loses table structure, page headers/footers bleed into content, and multi-column text gets interleaved incorrectly.

Choose and test chunking strategy

Experiment with multiple chunking approaches: fixed-size (500-1000 tokens), recursive character splitting, semantic chunking (split by meaning), or document-structure-aware chunking (by sections/headers). Test at least 3 strategies.

Why it matters: Chunk size and boundaries directly determine retrieval precision. Too small and you lose context. Too large and you dilute relevance. Wrong boundaries and you split critical information across chunks.

Common pitfall: Picking a chunk size without testing. The optimal size depends on your data -- legal documents may need 1500-token chunks to capture full clauses, while FAQ entries may work best at 200 tokens.

Add metadata to every chunk

Attach structured metadata: source document name, section title, page number, creation date, document type, and any domain-specific tags. Store this alongside the vector embedding.

Why it matters: Metadata enables filtered retrieval ("only search in legal documents from 2024"), source attribution in answers, and debugging. Without metadata, you cannot trace an answer back to its source.

Common pitfall: Adding metadata as an afterthought. Retrofitting metadata onto millions of already-indexed chunks is painful. Design the schema upfront.

Handle document updates and deletions

Build an incremental processing pipeline. Track document versions, detect changes, re-chunk and re-embed only modified documents, and remove stale chunks from the index.

Why it matters: Real document corpora change constantly. Without an update mechanism, your RAG system answers with outdated information, which is worse than no answer at all.

Common pitfall: Re-indexing the entire corpus on every update. This is slow, expensive, and causes temporary inconsistencies. Use content hashing to detect actual changes.

2

Embedding & Indexing

Select and benchmark embedding models

Test at least 3 embedding models on your actual data. Consider: OpenAI text-embedding-3-large, Cohere embed-v3, open-source options (BGE, E5, GTE). Benchmark on retrieval precision against a test set of queries.

Why it matters: Embedding model choice can make a 20-30% difference in retrieval quality. General-purpose embeddings may underperform domain-specific ones for technical or specialized content.

Common pitfall: Choosing based on MTEB leaderboard scores alone. Leaderboard performance does not always transfer to your domain. Always test on your own query-document pairs.

Choose and configure vector database

Evaluate options based on your scale: Chroma or Qdrant for prototyping, Pinecone or Weaviate for managed production, pgvector for staying within PostgreSQL. Configure the index type (HNSW for speed, IVF for memory efficiency).

Why it matters: The vector database is your system's backbone for retrieval speed and accuracy. Wrong index configuration leads to either slow queries or missed relevant results.

Common pitfall: Using default index settings in production. HNSW parameters (ef_construction, M) dramatically affect recall vs. speed trade-offs. Tune them for your dataset size and latency requirements.

Implement hybrid search (BM25 + vector)

Combine keyword-based search (BM25/TF-IDF) with semantic vector search. Use reciprocal rank fusion (RRF) or learned weights to merge results. This catches both exact keyword matches and semantic similarities.

Why it matters: Pure vector search misses exact term matches (product IDs, error codes, names). Pure keyword search misses paraphrases. Hybrid search consistently outperforms either alone by 10-25% in benchmarks.

Common pitfall: Equal weighting of BM25 and vector scores. The optimal ratio depends on your query types -- technical queries with specific terms benefit from heavier BM25 weighting.

3

Retrieval Strategy

Implement query preprocessing

Transform user queries before retrieval: expand abbreviations, correct spelling, resolve pronouns using conversation history, and optionally generate multiple query variants (HyDE or multi-query) for broader recall.

Why it matters: Users write queries differently from how documents are written. "How do I fix error 404?" and "HTTP 404 Not Found resolution steps" are the same intent but match different chunks. Query preprocessing bridges this gap.

Common pitfall: Over-transforming queries and losing the original intent. Always keep the original query as one of the search variants and merge results.

Add a re-ranking stage

After initial retrieval (top-20 to top-50 candidates), apply a cross-encoder re-ranker (e.g., Cohere Rerank, BGE Reranker) to re-score and re-order results based on query-document relevance. Then take the top-5 to top-10.

Why it matters: Bi-encoder embeddings (used for initial retrieval) sacrifice accuracy for speed. Cross-encoder re-rankers see the query and document together, giving much more accurate relevance scores. This typically improves precision by 15-30%.

Common pitfall: Re-ranking too few candidates. If you only retrieve top-5 and re-rank those, the re-ranker cannot surface relevant documents that were ranked 6th-20th in initial retrieval.

Implement contextual chunk enrichment

When a chunk is retrieved, also fetch its surrounding context: the parent document section, adjacent chunks, or a document summary. Include this additional context in the LLM prompt to prevent the "lost context" problem.

Why it matters: A chunk that says "As shown in Table 3 above, the results improved by 40%" is useless without Table 3. Contextual enrichment ensures the LLM has enough information to give a complete, accurate answer.

Common pitfall: Including too much context and exceeding the context window or diluting relevance. Use a hierarchy: chunk first, then section, then summary -- and trim based on available token budget.

Set up metadata filtering

Allow retrieval to be scoped by metadata: date ranges, document types, categories, or access permissions. Implement pre-filtering (before vector search) for large exclusions and post-filtering for fine-grained control.

Why it matters: Without filtering, a query about "2024 revenue" might retrieve 2022 data that happens to be semantically similar. Metadata filters enforce hard constraints that semantic similarity alone cannot guarantee.

Common pitfall: Pre-filtering too aggressively and eliminating all candidates. Always monitor the number of candidates remaining after filtering and fall back to broader search if needed.

4

Generation

Design the context injection prompt

Craft a system prompt that instructs the LLM to answer based on provided context, cite sources, and explicitly state when the context does not contain sufficient information. Structure retrieved chunks with clear delimiters and source attribution.

Why it matters: The prompt template determines whether the LLM faithfully uses retrieved context or falls back to its parametric knowledge (hallucination). A well-designed prompt reduces hallucination rates by 50% or more.

Common pitfall: Placing all context at the beginning of a long prompt. LLMs exhibit "lost in the middle" effects -- information in the middle of the context gets less attention. Place the most relevant chunks at the start and end.

Implement source citation

Require the LLM to cite specific sources for each claim in its response. Map citations back to original documents with page numbers or section references. Display citations to users for verification.

Why it matters: Citations enable users to verify answers, build trust, and quickly access the original document for more detail. Without citations, users have no way to distinguish accurate answers from hallucinations.

Common pitfall: Asking for citations without validating them. LLMs sometimes generate plausible-sounding but non-existent citations. Always validate that cited sources actually exist and contain the claimed information.

Handle "I don't know" gracefully

Detect when retrieved context is insufficient (low relevance scores, no matching chunks) and instruct the LLM to say "I don't have enough information to answer this" rather than guessing. Provide suggested alternative queries.

Why it matters: A system that confidently gives wrong answers is worse than one that admits uncertainty. Users need to trust that when the system answers, it is drawing from actual source material.

Common pitfall: Setting the relevance threshold too high (never answers) or too low (always answers, even from irrelevant context). Calibrate the threshold using a labeled test set of in-scope and out-of-scope queries.

Enable streaming responses

Stream the LLM response token-by-token to the user interface rather than waiting for the complete response. Display a loading state during the retrieval phase, then stream the generation.

Why it matters: Time-to-first-token is the most important latency metric for user experience. A 5-second wait feels much shorter when tokens start appearing after 500ms. Streaming reduces perceived latency by 3-5x.

Common pitfall: Not handling streaming errors gracefully. If the stream breaks mid-response, users see a truncated answer with no indication of failure. Implement retry logic and clear error states.

5

Evaluation

Build a golden test dataset

Create 50-100+ question-answer pairs with source document references. Include diverse query types: factual lookups, multi-hop reasoning, temporal queries, out-of-scope queries, and adversarial inputs. Have domain experts validate the ground truth answers.

Why it matters: Without a test dataset, every change is a guess. You cannot know if a new chunking strategy, embedding model, or prompt template actually improves your system. This is the foundation of data-driven iteration.

Common pitfall: Creating test queries that are too easy or too similar. Ensure diversity: include queries with typos, queries that span multiple documents, queries about information that is NOT in your corpus, and ambiguous queries.

Measure retrieval quality separately

Evaluate retrieval independently from generation using information retrieval metrics: Precision@k, Recall@k, Mean Reciprocal Rank (MRR), and NDCG. This tells you if the right chunks are being found, regardless of how the LLM uses them.

Why it matters: If retrieval fails, generation cannot succeed. By measuring retrieval separately, you know whether to fix your indexing/search or your prompt/generation. Most RAG failures are retrieval failures.

Common pitfall: Only measuring end-to-end answer quality. If the final answer is wrong, you cannot tell whether the retriever failed to find the right chunk or the generator failed to use it correctly.

Measure generation quality

Evaluate generation using RAGAS metrics: faithfulness (does the answer stick to the context?), answer relevancy (does it address the question?), and context precision (was the right context used?). Run LLM-as-judge evaluations for nuanced quality assessment.

Why it matters: Even with perfect retrieval, the LLM can hallucinate, miss key points, or give verbose irrelevant answers. Generation metrics catch these failure modes and guide prompt optimization.

Common pitfall: Relying only on automated metrics. LLM-as-judge evaluations can be biased. Periodically validate automated scores against human judgments to ensure your metrics are calibrated correctly.

6

Production Deployment

Add observability and tracing

Instrument every step: query preprocessing, retrieval (which chunks, scores), re-ranking, context assembly, and generation. Use tools like LangSmith, Langfuse, or Arize Phoenix. Log latency, token counts, and costs per request.

Why it matters: Production debugging without tracing is guesswork. When a user reports a bad answer, you need to see exactly what was retrieved, what context was sent to the LLM, and what the model generated to diagnose the root cause.

Common pitfall: Logging too little (cannot diagnose) or too much (storage costs explode, PII leakage risk). Log structured traces with the ability to sample at lower rates for high-traffic systems.

Implement caching

Add multi-level caching: exact-match cache (identical queries), semantic cache (similar queries using embedding similarity), and embedding cache (avoid re-embedding the same document chunks). Set appropriate TTLs for each cache level.

Why it matters: In production, 30-60% of queries are repeats or near-repeats. Caching can reduce LLM costs by 40-50% and improve latency from seconds to milliseconds for cache hits.

Common pitfall: Caching answers to queries that depend on user context or permissions. If User A's answer is served to User B from cache, you may leak private information. Cache keys must include access-control context.

Set up monitoring and alerting

Monitor: retrieval latency (p50, p95, p99), generation latency, error rates, cache hit rates, average relevance scores, token usage, and costs. Set alerts for anomalies: sudden drops in relevance scores or spikes in hallucination rates.

Why it matters: Production systems degrade silently. A corpus update might corrupt an index, a model API might start returning errors, or query patterns might shift. Without monitoring, you only learn about problems from angry users.

Common pitfall: Alerting on every metric deviation. Alert fatigue is real. Focus alerts on user-facing impact: answer quality drops, latency spikes above user tolerance, and error rate increases.

Implement user feedback collection

Add thumbs up/down buttons, allow users to flag incorrect answers, and collect optional text feedback. Store feedback linked to the full trace (query, retrieved chunks, generated answer) so you can analyze failure patterns.

Why it matters: User feedback is the ultimate evaluation signal. It tells you what matters to real users, reveals failure modes you did not anticipate in your test set, and provides data for continuous improvement.

Common pitfall: Collecting feedback but never analyzing it. Schedule a weekly review of negative feedback, categorize failure modes, and prioritize fixes based on frequency and severity.