Automate retrieval-augmented generation testing with clear, measurable steps. This DeepEval LLM evaluation guide shows how to build a repeatable pipeline that checks retrieval quality, grounds answers in context, and uses an LLM-as-a-judge to score responses. You get actionable metrics (precision, recall, faithfulness) plus a simple fallback so evaluations run even without an API key. Large language model apps break in subtle ways. Answers can sound good yet ignore the question. Retrieval can miss key facts. Hallucinations can hide in smooth text. You need tests that catch these issues before they reach users. This article gives you a practical path to automate RAG quality checks with DeepEval, a custom TF-IDF retriever, and LLM-as-a-judge metrics. It mirrors how software teams write unit tests and turns evaluation into a repeatable workflow. You will see how to set up the environment, define a small gold dataset, generate answers, and score them with clear metrics.

What you will build

You will build a simple but strong evaluation loop for RAG:

Create a tiny knowledge base of short documents.

Use a TF-IDF retriever to fetch top-k context chunks for a query.

Generate an answer with OpenAI or an offline extractive fallback.

Package each run into DeepEval test cases.

Score with faithfulness, contextual precision, contextual recall, answer relevancy, and a G-Eval rubric.

Aggregate scores and judge explanations to find and fix weak spots.

This approach scales from a notebook to CI. You can gate releases on metric thresholds, compare retrievers, and spot regressions early.

DeepEval LLM evaluation guide: key building blocks

This DeepEval LLM evaluation guide focuses on building blocks that reduce noise, increase repeatability, and keep costs under control.

Stable environment setup

Pin common dependencies. In many notebooks, pinning NumPy and reinstalling key libraries avoids runtime conflicts. Install deepeval, scikit-learn, pandas, and tqdm. Optionally set an OpenAI API key via a secure input so the same notebook runs online or offline.

Pin NumPy if your notebook environment is unstable.

Install deepeval and its dependencies.

Store the OpenAI API key in an environment variable only if present.

This keeps your evaluation script predictable across machines and collaborators.

Curate a small gold dataset

Start with 5–10 focused questions and short reference snippets. Each item should include:

Query: a direct, user-like question.

Expected output: a compact, correct answer (the “gold”).

Documents: short passages that hold the needed facts.

Use clean, self-contained texts. Avoid vague phrasing. Include a pitfall case, such as a question where expected_output is required by certain metrics. This catches missing data errors early.

A simple custom TF-IDF retriever

A TF-IDF retriever is easy to understand and fast to debug. Use unigrams and bigrams. Concatenate title and text for each doc. For each query:

Vectorize the query with the same Tf-idfVectorizer.

Compute cosine similarity to all doc vectors.

Select the top-k matches and keep scores and texts.

This baseline is not perfect, but it is stable and reproducible. You can swap it later for an embedding-based retriever without changing the evaluation flow.

Generation with a safe fallback

Use a two-path generation strategy:

Primary: call an LLM (for example, OpenAI) with a RAG prompt that says “Use only the provided context. If the answer is not in the context, say you don’t know.”

Fallback: extract key sentences from the retrieved text using keyword matching. This keeps the pipeline working when the API is unavailable.

Keep the retrieval context separate from the answer text. DeepEval will use the raw context for RAG metrics, no matter how you generate the final answer.

DeepEval test cases

Each test case should include:

input: the user’s query.

expected_output: the gold answer (needed by some metrics).

actual_output: the model or fallback answer.

retrieval_context: the ranked list of retrieved passages.

Store your test cases in a list. This allows batch evaluation and clean aggregation of scores.

Metrics that matter

The following metrics give you clear signals about your system:

Faithfulness: Does the answer stay grounded in the retrieved context? This catches hallucinations.

Contextual Precision: Are the most relevant chunks ranked near the top? This shows how well your retriever orders results.

Contextual Recall: Did the retriever bring back enough relevant context to answer the question? Low recall points to missing facts.

Answer Relevancy: Does the answer address the query? This catches answers that are correct but off-topic.

Contextual Relevancy: Are the retrieved chunks on-topic for the question?

G-Eval (GEval): A customizable LLM-as-a-judge rubric. Define criteria like correctness, clarity, tone, or policy compliance in plain language.

Together, these measures paint a full picture of retrieval and generation quality.

Step-by-step workflow

Below is a simple flow you can adapt to your stack.

1) Retrieve context

For each query, call your TF-IDF retriever with k=4 or k=5. Save both the text and the similarity scores. Keep the list ordered by relevance.

2) Generate an answer

If an API key is present:

Construct a prompt that includes the query and numbered context blocks.

Set temperature low (for example, 0.2) to reduce randomness.

Ask the model to say “I don’t know” if the context does not contain the answer.

If the API fails or is not available, run your extractive fallback. It selects sentences that share keywords with the query and forms a short answer.

3) Package into test cases

For each query:

input = the query text.

expected_output = the gold answer from your dataset.

actual_output = the generated or extracted answer.

retrieval_context = the ranked passages from the retriever.

Create a list of these test cases. This is your evaluation batch.

4) Define metrics

Instantiate the DeepEval metrics you need. For G-Eval, write a short rubric in plain language. For example:

Correctness: Does the answer match the facts in the context?

Grounding: Does the answer avoid claims not supported by the context?

Clarity: Is the answer concise and easy to read?

Set thresholds if you plan to fail a build when scores drop.

5) Run evaluate() and collect results

Run the DeepEval evaluate function on your test cases and metrics. The tool will call the LLM judge as needed, compute RAG scores, and return structured results. Convert them to a DataFrame for sorting, filtering, and plotting.

Sort by lowest faithfulness to see likely hallucinations first.

Sort by lowest recall to find missing documents or bad chunking.

Read the judge’s explanations to understand why a score is low.

Interpreting scores and fixing issues

Numbers are only useful if they guide action. Here is how to read common patterns and what to try next.

Low faithfulness

When faithfulness is low, the answer likely includes unsupported claims.

Ask the model to cite context markers (e.g., [CTX 2]).

Lower temperature and shorten max tokens to reduce drift.

Strip non-relevant context to avoid confusion.

Improve prompt instructions: “Use only the provided context.”

Low contextual recall

When recall is low, the retriever did not bring back needed facts.

Increase k and test again.

Add synonyms or domain terms to queries before retrieval.

Switch to embeddings and try different models.

Fix chunk sizes and overlaps so facts stay intact.

Low contextual precision

When precision is low, the retriever ranks weak passages too high.

Tune TF-IDF n-grams or switch to BM25 for sparse retrieval.

Use query expansion carefully; avoid adding noise.

Try hybrid retrieval (sparse + dense) and reranking.

Low answer relevancy

The answer might be correct but not responsive to the question.

Improve the system prompt: “Answer only the asked question.”

Shorten answers unless the question asks for detail.

Enforce a structure: a one-sentence answer plus one supporting quote.

Missing expected_output pitfalls

Some metrics need expected_output. If it is missing, the evaluation can fail or produce weak signals.

Always include expected_output for tests that check precision/recall.

Keep gold answers short and specific to avoid ambiguity.

Hardening the pipeline for production

Once your notebook gives useful scores, make it a guardrail in your release process.

Budget-aware runs

Judge models cost money. Keep costs stable:

Evaluate on a small, representative test set per PR.

Run a larger suite nightly or weekly.

Cache judge outputs keyed by (query, context, answer, rubric) to avoid re-scoring unchanged cases.

CI integration

Add evaluation to CI:

Fail the build if average faithfulness falls below a threshold.

Post a summary comment with top 3 failing cases.

Store history to track trends over time.

Regression gates and drift monitoring

Your knowledge base, retriever, and model may change.

Lock versions for models and retrievers unless you plan an update.

Compare before/after scores when changing embeddings or chunking.

Watch for data drift: run periodic tests on fresh questions and known edge cases.

Extensions and advanced ideas

Once the basics are solid, try these upgrades.

Swap retrievers

Replace TF-IDF with embeddings and reranking. Keep the same DeepEval tests so you compare apples to apples. Try different embedding models and top-k values. Evaluate cost vs. gain.

Multi-hop questions

Add questions that require two or more facts. Update your rubric to reward correct chaining and penalize leaps beyond the context.

Multilingual tests

If your users ask in multiple languages, include those queries. Add language-specific stop words to TF-IDF or use multilingual embeddings.

Safety and policy rubrics

Use G-Eval to check tone, compliance, or sensitive content. Define what is allowed and what is not. Score answers across safety checks before deployment.

Judge reliability and calibration

LLM judges can vary. Improve reliability:

Use two judges and average scores or flag disagreements.

Include a few anchor cases with known labels to calibrate each run.

Keep the rubric short and concrete; avoid vague criteria.

Determinism and seeds

Lower temperature for both generator and judge. Where possible, set seeds. Smaller variance helps you detect true changes, not random swings.

Practical tips and gotchas

Keep answers short. Long answers inflate cost and risk drift.

Normalize text (case, whitespace) before comparing or scoring.

Log everything: query, context IDs, scores, and judge rationales.

Chunk documents by meaning, not fixed length only.

Handle “I don’t know” as a valid answer when context is missing.

Balance your test set: easy wins, near misses, and tough edge cases.

Review 3–5 failing cases by hand each week to guide improvements.

Why this approach works

RAG systems have two moving parts: getting the right context and saying the right thing about it. Retrieval metrics (precision, recall, contextual relevancy) tell you if you fetched the right facts. Generation metrics (faithfulness, answer relevancy, custom G-Eval) tell you if you used those facts correctly. Packaging everything as test cases lets you catch problems early, measure improvements, and avoid silent regressions. This is the heart of this DeepEval LLM evaluation guide: make quality visible, measurable, and actionable.

Putting it all together

Start small. Build a five-question gold set. Add a TF-IDF retriever and a strict RAG prompt. Plug the pieces into DeepEval metrics. Run the evaluate function and read the judge explanations. Fix one issue at a time: improve chunking, tweak prompts, or adjust top-k. Add a few more tests. When scores stabilize, wire the run into CI and set a pass threshold. Over time, swap components—embeddings, rerankers, more detailed rubrics—without losing your baseline. A strong evaluation loop turns guesswork into engineering. It helps your team ship changes with confidence, reduce hallucinations, and keep answers on target. Use this DeepEval LLM evaluation guide to make RAG testing systematic, fast, and trustworthy from day one—and keep it that way as your product grows. In short, follow this DeepEval LLM evaluation guide to set up stable tests, pick clear metrics, and use LLM-as-a-judge scoring to monitor retrieval and generation quality. Your users will see better answers, and your team will ship with fewer surprises. (Source: https://www.marktechpost.com/2026/01/25/a-coding-implementation-to-automating-llm-quality-assurance-with-deepeval-custom-retrievers-and-llm-as-a-judge-metrics/) For more news: Click Here

FAQ