RAG fundamentals for Data Science interviews

Why RAG keeps coming up

Your interviewer opens with "we have 40,000 internal docs and want an assistant our support team can actually trust — walk me through how you'd build it." That is a RAG (Retrieval-Augmented Generation) question, and it shows up in roughly every applied DS / ML loop at Notion, Stripe, Linear, Anthropic and most AI-forward orgs. The reason is simple: an LLM only knows what was in its training data, and fine-tuning a 70B model on a moving corporate corpus is wasteful, slow, and stale within a quarter.

RAG flips the problem. Instead of baking knowledge into weights, you keep the corpus on the side, retrieve the most relevant chunks at query time, and inject them into the prompt. The model stays general; the knowledge stays fresh; you can show source citations and cut hallucinations to a manageable level. That last part — citations and grounded answers — is why legal, healthcare, and finance teams will only ship LLM features on top of a RAG pipeline.

This post covers the components, where they break, and what interviewers listen for.

High-level architecture

Every production RAG system has the same skeleton. There are two phases, and you should be able to draw the diagram on a whiteboard in under a minute.

[query] → embed → search → top-K docs → LLM (with context) → answer
                    ↑
              vector DB
                    ↑
[docs] → chunk → embed → index

The indexing phase is offline: raw documents → chunks → dense vectors → vector DB (with original text). The retrieval and generation phase is online: embed the query with the same model, run nearest-neighbor search for the top-K chunks, stuff them into a prompt template, let the LLM compose a grounded answer.

The skeleton looks trivial. The reason RAG is a real job for an applied DS is that every arrow is a tuning surface with five plausible choices and one that works for your corpus. Chunk size, overlap, embedding model, index type, top-K, prompt, rerank or not, hybrid or pure — change any one and faithfulness moves 5-10 points.

Chunking

Chunking determines your retrieval ceiling. You split each document into smaller units; each unit becomes a row in the vector store. If the chunk is too big, the embedding becomes a smeared average and precision collapses. If it is too small, you lose surrounding context and the LLM answers without enough signal.

The four patterns to name in an interview are fixed-size, sentence-based, semantic, and structure-aware chunking. Fixed-size — 256 to 512 tokens with 10-20% overlap — is the baseline everyone defaults to and works well for prose. Sentence-based splitters use a tokenizer to avoid mid-sentence cuts, then pack sentences to a token budget. Semantic chunking uses an embedding model to find topic shifts and split there; most accurate, slowest. Structure-aware chunking respects markdown headings, HTML tags, or code blocks — for technical docs this is usually the best answer.

For most enterprise corpora, start with chunk_size=384, chunk_overlap=64, structure-aware splitter, and only move off those defaults when your eval set tells you to.

Embeddings and vector search

Each chunk gets embedded into a dense vector — typically 384 to 3072 dimensions — and stored in a vector DB with the original text and metadata. At query time you embed the query with the same model and run a similarity search, usually cosine.

The credible 2025-2026 embedding options are OpenAI text-embedding-3-large and text-embedding-3-small, open-source BGE and E5 families, Cohere managed embed, and Voyage for domain-specific work. Pick the smallest model that hits your eval threshold — index size grows with dimension.

For the vector DB, the decision tree is short: in-memory FAISS for prototypes and sub-million corpora; pgvector inside an existing Postgres if you want one fewer service; Qdrant, Weaviate, Pinecone, or Milvus for anything bigger. Snowflake and Databricks both now ship native vector support if you're already on their stack.

from openai import OpenAI
client = OpenAI()

def embed(text: str) -> list[float]:
    resp = client.embeddings.create(
        input=text,
        model="text-embedding-3-small",
    )
    return resp.data[0].embedding

# offline indexing
for chunk in chunks:
    vec = embed(chunk["text"])
    vector_db.upsert(
        chunk_id=chunk["id"],
        vector=vec,
        metadata={"source": chunk["source"], "ts": chunk["ts"]},
    )

# online retrieval
query_vec = embed(user_query)
hits = vector_db.search(query_vec, top_k=10, filter={"source": "docs"})

On a corpus of millions of vectors, an exact nearest-neighbor scan is too slow. ANN indexes — HNSW and IVF being the two you should be able to compare — give you sub-50ms latency at the cost of a few recall percentage points. HNSW is the default for read-heavy small-to-medium indexes; IVF with product quantization wins when memory is tight or the corpus is enormous.

Hybrid retrieval and reranking

Pure vector search has a known failure mode: it loses on keyword-exact queries. If a user asks about a specific error code, SKU, statute number, or function name, semantic similarity will happily return chunks that are "about" that topic without containing the exact token, and your answer will be wrong.

The fix is hybrid retrieval. You run two retrievers in parallel — a dense vector search and a lexical search like BM25 — and fuse the results with Reciprocal Rank Fusion (RRF) or a learned combiner. RRF is the workhorse: it just sums 1/(k+rank) across rankers, requires zero training, and almost always beats either retriever alone.

Reranking sits on top of retrieval. The retriever's job is recall — get the right chunk into the top 50 — and the reranker's job is precision: reorder those 50 so the best 3-5 land at the top. A cross-encoder reads the query and the chunk together and produces a relevance score, which is more accurate than the bi-encoder used for the first stage but 10-100x more expensive per pair.

vector + BM25 → top-50 → reranker (cross-encoder) → top-5 → LLM

The standard rerankers in 2026 are Cohere Rerank, BGE Reranker, and Voyage's rerank-2 model. On most benchmarks, adding a reranker on top of hybrid retrieval moves recall@5 by another 8-15 points and is the single highest-ROI change in a tuned pipeline.

Evaluating a RAG system

Evaluating RAG is harder than classification. There is no single label per query — the system can retrieve the right context and still hallucinate, or retrieve garbage and still produce a fluent-sounding answer. Measure retrieval and generation separately, then together.

For retrieval, recall@K (gold chunk in top-K?) and MRR (position of first relevant chunk) are the standard pair. Even 100-300 labeled query-to-chunk pairs is enough to start tuning chunk size and top-K.

For generation, RAGAS is the de-facto framework. It uses LLM-as-judge to score faithfulness, answer relevancy, and context relevancy. LLM-as-judge is noisy — calibrate against a small human-labeled set of 50-100 examples before trusting absolute numbers.

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_relevancy

results = evaluate(
    dataset,
    metrics=[faithfulness, answer_relevancy, context_relevancy],
)

Human evaluation remains the gold standard for shipping decisions. Run it on a frozen 50-100-example test set per major release. Use RAGAS for the fast inner loop (chunk size, top-K) and human review for the ship/no-ship decision.

Common pitfalls

The most common first-time mistake is defaulting to chunk_size=1000 because the LangChain quickstart used that number. A thousand tokens is too coarse for most embedding models — the vector becomes an average of multiple topics and top-K precision tanks. Run a chunk-size sweep on a small eval set; almost everyone lands between 256 and 512 tokens with 10-20% overlap.

A close second is shipping pure vector search with no lexical fallback. The day a user searches for an exact error code or customer ID, your retriever returns five chunks that are "semantically about errors" and zero that contain the actual code. Hybrid retrieval with BM25 plus dense, fused with RRF, is two lines of config in most vector DBs and there is rarely a good reason to skip it.

A more subtle pitfall is ignoring metadata filters. Your corpus has dates, authors, document types, regions; if you embed the question and search across all of it, you're asking the model to disambiguate using vectors alone. A where clause on date range or document type before the vector search frequently doubles precision at top-5 and costs nothing at query time.

Teams also routinely skip reranking because it adds 100-300ms of latency. That is the wrong tradeoff in almost every case where quality matters. A cross-encoder rerank on the top-50 moves recall@5 noticeably and is the highest-ROI change on a tuned pipeline. If latency budget is tight, run reranking only on long-tail queries where retriever confidence is low.

The most dangerous failure mode is not measuring faithfulness at all. A RAG system that retrieves perfectly can still hallucinate confidently, and without faithfulness scoring you will not know it until a customer screenshots a wrong answer. Wire up RAGAS or an equivalent LLM-as-judge metric in CI from day one.

Finally, treating RAG as a single retrieval pass breaks for compound questions. "Compare our refund policy for EU and US customers" needs at least two retrievals. Query decomposition, multi-hop retrieval, and agentic patterns like ReAct are the next layer once single-hop is solid.

Related reading

• Embeddings for DS interviews
• Embedding quality evaluation
• Transformer architecture for DS interviews
• Attention mechanism for DS interviews
• Hallucinations and LLM evals
• RAG evaluation deep dive
• Long-context LLMs

If you want to drill these RAG patterns against realistic interview questions, NAILDD ships with a growing bank of applied ML and DS prompts built around exactly this stack.

FAQ

When should I use RAG vs fine-tuning?

Reach for RAG when the corpus changes often, you need source citations, or you want to control hallucinations on factual content. Reach for fine-tuning when you need a stable change to style, format, or behavior — a brand voice, a structured output schema, a domain dialect — and the underlying knowledge is reasonably static. A common production pattern is fine-tuning a small model on output format and using RAG for the actual knowledge. The interviewer is checking that you understand fine-tuning teaches behavior, retrieval supplies facts.

Which LLM should I pair with my RAG pipeline?

For quality-sensitive workloads, frontier models — GPT-class, Claude Sonnet-class — are still the default. For latency or cost-sensitive use cases, smaller models like Claude Haiku, GPT-4o-mini, or open-source Llama 3.1 8B work well once retrieval is strong, because the generation step becomes mostly summarization. Better retrieval lets you use a cheaper generator without losing quality, which is why teams invest in retrieval before reaching for the biggest model.

Does hybrid search work in Pinecone, Qdrant, and pgvector?

Yes, all three support it in 2026. Pinecone and Qdrant ship native sparse-plus-dense indexes with server-side RRF. pgvector requires a separate tsvector column for BM25 and combining ranks in SQL — that is fine and many teams ship it that way. Weaviate and Milvus also expose hybrid out of the box. The interviewer cares that you know hybrid is a config flag, not a research project.

How do I actually reduce hallucinations?

Stack three things: a strict system prompt that says "answer only from the provided context, otherwise say you don't know," a citation requirement forcing the model to point at specific chunks, and a faithfulness eval in CI that catches regressions before they ship. Stacked together they cut hallucinations by a large factor on most benchmarks. For high-stakes domains, add a second-pass verifier model that re-reads the answer against the context.

How many chunks should I pass to the LLM?

Start with top-5 after reranking and tune from there. Past 10 chunks you usually see hallucinations rise because the model picks up irrelevant context, and you burn token budget. Long-context models change this calculus — a 200k or 1M-token model can absorb more chunks, but recall@K still matters more than K itself. The right answer is "as few as possible to get the right context, and no more."