RAG Engineer

A RAG engineer specialises in retrieval-augmented generation: building the pipelines that ingest documents, create embeddings, populate vector stores, retrieve the right context at query time, and evaluate whether the system is actually returning accurate, grounded answers. It is the role that makes LLMs reliable over private knowledge.

What a RAG engineer does day-to-day

RAG engineers own the pipeline that connects an LLM to a knowledge base. The job involves three distinct phases: ingestion (getting documents into the system in a form the retriever can work with), retrieval (finding the right content at query time), and evaluation (measuring whether the system is actually helping users get accurate answers).

In practice, most of the iteration time is in retrieval quality. A RAG engineer will spend a week improving chunking strategy, another tuning the embedding model choice, another adding a reranker, and another building the eval harness to measure whether any of it helped. The work is empirical: form a hypothesis, run an experiment, measure against a held-out eval set.

Skills companies expect

RAG engineering is a specialisation within LLM engineering. Companies hiring for this specific title expect hands-on experience with the full retrieval stack - not just familiarity with LangChain wrappers, but genuine understanding of what happens when retrieval quality is poor and how to fix it.

The interview process

RAG engineering interviews typically run 3-4 rounds. They lean heavily on system design and applied knowledge, with coding focused on retrieval pipeline implementation rather than algorithmic puzzles.

1. 1

   Recruiter and technical screen

   Round 130-45 min

   Background in LLMs and RAG, and a warm-up on the retrieval stack: what chunking strategies do you know, how does vector search work, what is a reranker and when would you use one? Some companies send a take-home - implement a small RAG pipeline over a document corpus and evaluate its quality.
2. 2

   RAG knowledge deep-dive

   Round 245-60 min

   Applied and analytical: how would you evaluate a RAG pipeline with no labelled data? What would cause retrieval to fail on a query where you can see the answer is in the corpus? How do you handle documents where the important information spans multiple pages? Walk me through hybrid search and how you would decide whether it improves results. These are open-ended - they want your reasoning process.
3. 3

   Coding round

   Round 360 min

   Implement a chunking function, write an embedding pipeline that handles batching and rate limiting, query a vector store with metadata filters, or build a minimal eval harness that computes retrieval precision at k against a labelled dataset. Python throughout - mid-level algorithmic problems occasionally appear.
4. 4

   RAG system design

   Round 460 min

   Design a production RAG system for a specific use case: an enterprise knowledge base, a legal document Q&A tool, or a technical support bot over a large documentation corpus. Cover ingestion, chunking, embedding, indexing, retrieval, reranking, generation, evals, and how the system handles incremental document updates without full re-indexing.

Common interview questions

Retrieval fundamentals

• What chunking strategies exist and when do you use each? Fixed-size with overlap is simple and predictable, good as a baseline. Recursive character text splitting respects sentence and paragraph boundaries, which usually produces better semantic coherence. Semantic chunking splits at embedding-dissimilarity boundaries - higher quality but slower to compute. Sentence-window chunking stores small units for retrieval but sends a larger surrounding window to the LLM, decoupling retrieval granularity from generation context. Choose based on document type, query pattern, and how much preprocessing cost you can afford.
• What is the difference between dense and sparse retrieval? Dense retrieval uses neural embeddings to find semantically similar documents - it generalises well across phrasings but can miss exact keyword matches. Sparse retrieval (BM25, TF-IDF) scores documents by term frequency and works well for exact keyword matches, product codes, and proper nouns. Hybrid search combines both via reciprocal rank fusion - in practice, it almost always outperforms either approach alone for enterprise search.
• How do you evaluate a RAG pipeline when you have no labelled data? Synthetic eval dataset generation: use an LLM to generate question-answer pairs from your documents, then use those as a test set. RAGAS synthetic test generation is the standard approach. It is not as good as human labels, but it gives you a measurable baseline immediately and catches obvious retrieval failures. Use it as a bootstrap until you can collect real user queries.
• What causes a RAG system to return a correct chunk but still hallucinate? The generation model is not faithfully using the retrieved context. Common causes: the context is too long and the model ignores content in the middle (lost-in-the-middle phenomenon), the system prompt does not explicitly instruct the model to answer from the context, or the model has strong prior beliefs that override the retrieved evidence. Measure faithfulness separately from answer relevancy with RAGAS.
• What is HNSW and why is it used in vector databases? Hierarchical Navigable Small World - a graph-based approximate nearest neighbour index. It provides sub-linear query time (typically O(log n)) at high recall, compared to O(n) for brute-force exact search. The tradeoff is memory and index-build time. At the scale most production RAG systems operate (millions of vectors), HNSW is the default choice. See Vector Databases for a full explanation.

Production and optimisation

• How do you handle incremental document updates without re-indexing everything? Store a document ID and version hash in the vector store as metadata. On ingestion, check whether the document has changed (hash comparison). Delete and re-embed only the chunks from changed documents. For very large corpora, maintain a separate delta index and periodically merge it into the main index to avoid query-time overhead from multi-index search.
• When does a reranker help and when is it not worth the latency? A reranker helps most when the query and the relevant document use different vocabulary (a weakness of embedding similarity) or when the corpus is large and the top-k from vector search has low precision. It is not worth the latency when the corpus is small (exact search is fast), when your embedding model is well-tuned to the domain, or when your latency SLO is tight (rerankers typically add 100-300ms). Measure NDCG@5 before and after - only add the reranker if it improves it.

RAG system design topics

RAG system design asks you to think through the complete pipeline from raw document to user-facing answer, including the operational concerns that most demos skip: incremental updates, access control, latency under load, and a continuous evaluation loop that catches quality regressions.

• Design an enterprise document Q&A system. Ingestion pipeline (PDF/Word parsing, chunking, embedding, metadata extraction), vector store (HNSW index, per-document access control via metadata filters), retrieval pipeline (hybrid search, reranker), generation (LLM with grounded prompt and citations), eval pipeline (RAGAS running on sampled queries nightly), and incremental re-ingestion for document updates.
• Design a RAG system for a legal document corpus where precision matters more than recall. Use smaller chunks with overlap for tight retrieval, a high-quality cross-encoder reranker, citation verification (check that every claim in the answer appears in the retrieved context), and a low temperature generation model. Add a mandatory human review step for any output used in a legal decision.
• Design a RAG eval framework from scratch. Synthetic question generation from documents, retrieval-only evaluation (context precision, context recall), generation evaluation (faithfulness, answer relevancy), end-to-end evaluation (answer correctness against ground truth), CI integration that blocks deployments on eval regressions, and a dashboard showing quality trends over time.
• Design an ingestion pipeline for 10 million documents. Batch processing with a message queue (Kafka or SQS) to handle ingestion jobs, parallel embedding with rate-limit management, idempotent chunk IDs to support re-ingestion, progress tracking and dead-letter queues for failed documents, and a staging index that gets swapped in after quality validation before it serves live traffic.

Coding expectations

RAG coding rounds focus on the plumbing of retrieval pipelines: parsing, chunking, embedding, querying, and evaluation. Algorithmic problems occasionally appear but rarely go beyond medium complexity.

• Chunking implementation: write a function that splits a long document into chunks of a given size with configurable overlap, respecting sentence boundaries. Handle edge cases (very short documents, documents with no sentence boundaries).
• Embedding pipeline: batch a list of text chunks into groups that respect the embedding API's token limit, call the embedding API with async concurrency and retry logic, and return a mapping from chunk ID to embedding vector.
• Vector store query with filtering: given a query embedding, retrieve the top-k chunks from a vector store where a metadata field (e.g. document_type) matches a filter value. Return the chunks ranked by similarity.
• Retrieval eval: given a list of (query, relevant_chunk_ids) pairs and a retrieval function, compute precision@k and recall@k across the dataset and return a summary with per-query breakdowns for the worst-performing cases.
• Reciprocal rank fusion: implement RRF to combine rankings from dense vector search and BM25 sparse retrieval into a single merged ranking.

Salary ranges by region

RAG engineering compensation closely tracks LLM engineering, with a slight premium at companies where the product's core value proposition is retrieval quality (legal tech, enterprise search, research tools). Figures below are approximate median total compensation.

30/60/90-day plan

1. 1

   First 30 days - baseline the existing system

   MeasureMonth 1

   Map the existing RAG pipeline end-to-end: ingestion, chunking, embedding model, vector store, retrieval strategy, and generation prompt. If there is no eval harness, build one immediately - even a synthetic one with 50 question-answer pairs is enough to establish a baseline. Identify the biggest quality gap: is it retrieval (the right chunks are not coming back) or generation (the right chunks are retrieved but the answer is wrong)?
2. 2

   First 60 days - improve retrieval quality

   ImproveMonth 2

   Run one structured improvement experiment per week: try a different chunking strategy, add metadata filters, switch to hybrid search, or add a reranker. Measure each change against your eval baseline - context precision and recall at k. Ship the change that shows the largest improvement with acceptable latency impact.
3. 3

   First 90 days - own the eval pipeline

   OwnMonth 3

   Build and own a production eval pipeline: automated nightly eval runs on a representative query set, RAGAS metrics tracked over time, CI integration that flags regressions before deployment, and a dashboard the team uses to make data-driven decisions about RAG changes. This is the signal that you can drive quality independently, not just implement changes.

Common mistakes RAG engineers make

• Using fixed-size chunking for everything. Fixed-size chunking is a reasonable starting point, not a final answer. It frequently cuts sentences mid-thought, separates a question from its answer when they appear on adjacent lines, and treats a one-line heading as a complete chunk. Invest in chunking strategy early - it has an outsized impact on retrieval quality.
• No retrieval evals before production. Shipping a RAG system without measuring retrieval quality is like deploying a model without offline evaluation. You will not know whether retrieval is the problem or generation is the problem, and you will not have a baseline to measure improvements against. Build evals before you start optimising.
• Not filtering by metadata before vector search. Vector similarity search over your entire corpus when only a subset of documents is relevant to the query (by user permissions, date range, or document type) is both slower and noisier than pre-filtering by metadata before the vector search. Most production vector stores support metadata filtering at query time - use it.
• Skipping reranking because it adds latency. A cross-encoder reranker adds 100-300ms of latency but frequently improves retrieval precision enough to materially improve answer quality. Measure the quality improvement before deciding the latency is not worth it. For many use cases, users would rather wait 300ms longer for a correct answer than get an instant wrong one.
• Treating the embedding model as a fixed constant. General-purpose embedding models are not always the best choice for your domain. A model fine-tuned on legal text will outperform text-embedding-3-large on a legal corpus. Evaluate 2-3 embedding models on a sample of your actual queries using retrieval metrics before committing to one.

RAG engineer vs related roles

Related guides: RAG Explained for the conceptual foundation - how retrieval-augmented generation works from first principles; Vector Databases for how the storage and indexing layer works under the hood; and Embeddings Explained for how text is converted into vectors that retrieval can operate on.