Retrieval-Augmented Generation (RAG)

RAG is the most common architecture in real generative-AI products: a chatbot over company documents, a support assistant, a policy Q&A tool. Interviewers want you to walk through the pipeline, explain each design choice and — most importantly — say how you would know if it works.

1. What is RAG and why is it used?

Retrieval-Augmented Generation fetches relevant information from your own data at question time and gives it to the model with the question, so the answer is grounded in current, private and citable sources. It reduces hallucinations, keeps knowledge up to date without retraining, and respects access control because you choose what to retrieve for each user.

2. Walk through a RAG pipeline.

Indexing (offline): load documents → clean them → split into chunks → create embeddings → store vectors and metadata in a vector index. Query time: embed the question (optionally rewrite it first) → retrieve the top chunks → rerank → put the best chunks in the prompt with instructions to answer only from them and cite sources → generate → return answer with citations.

def answer(question: str, user) -> str:
    hits = vector_store.search(embed(question), top_k=20, filter={"allowed_roles": user.roles})
    best = rerank(question, hits)[:5]
    context = "\n".join(f'<doc id="{h.id}" source="{h.source}">{h.text}</doc>' for h in best)
    prompt = (f"<docs>{context}</docs>\n\nAnswer using only the docs above and cite doc ids. "
              f"If they do not contain the answer, say so.\n\nQuestion: {question}")
    return llm(prompt)

3. How do you choose a chunking strategy?

Chunks must be small enough to be precise and large enough to make sense alone. Split on natural boundaries (headings, paragraphs, sections) rather than fixed character counts, keep a modest overlap, and attach metadata (title, section, date, URL). Adding a short summary of where the chunk sits in its document (“contextual” chunks) noticeably improves retrieval. Tune size using your evaluation set.

4. What is a vector database? Do you always need one?

A store optimised for nearest-neighbour search over embeddings, usually with approximate indexes such as HNSW — examples include pgvector (PostgreSQL), Pinecone, Weaviate, Qdrant, Milvus and Elasticsearch/OpenSearch. For a few thousand chunks, pgvector or even an in-memory index is plenty; you do not need a dedicated service until scale or features demand it.

-- PostgreSQL with pgvector
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE chunks (id bigserial PRIMARY KEY, source text, content text, embedding vector(1024));
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);

SELECT id, source, content
FROM chunks
ORDER BY embedding <=> :query_embedding
LIMIT 10;

5. What is hybrid search and reranking?

Hybrid search combines keyword search (BM25) with vector search, because keywords catch exact terms such as product codes, error numbers and names that embeddings can miss. Results are merged (for example with reciprocal rank fusion). A reranker — a model that scores each (question, chunk) pair — then reorders the top candidates, usually improving precision significantly.

6. How do you evaluate a RAG system?

Evaluate the two halves separately. Retrieval: for a set of questions with known relevant documents, measure recall@k — did the right chunk appear in the top results? Generation: measure faithfulness (is every claim supported by the retrieved text?), answer relevance and correctness, using human review plus an LLM grader. Track “I don't know” rates too — refusing when the answer exists is also a failure.

7. RAG vs fine-tuning vs long context — when to use which?

RAG for knowledge that is large, changes often or needs citations and access control. Long context (pasting whole documents) when the material is small enough and you want simplicity — often combined with prompt caching. Fine-tuning to teach a style, format or narrow skill, not facts. Many products combine RAG with good prompting and never fine-tune.

8. Your RAG bot gives wrong answers. How do you debug it?

Look at one failure end to end. Was the right document in the index? Was it chunked badly? Did retrieval find it (check the top-k)? Did the reranker drop it? Was it in the prompt but ignored or misread? Fix the earliest broken step — most RAG problems are retrieval problems, not model problems.