Faithfulness and answer relevance judge against different references

A system that searches a company's documents for passages relevant to a question and asks a language model to answer from them can fail in more than one way, and the ways do not share a fix. The model can assert something none of the passages support. It can also summarise a passage perfectly and answer a question nobody asked, because search returned a plausible neighbour of the right document. Both produce a reply that reads well, and a single quality score covering both cannot say which one happened.

That arrangement — passages fetched from a collection, a model writing an answer out of them — is retrieval-augmented generation, or RAG, which topic 9.2 built by hand. A RAG-specific metric is a score aimed at one of its ways of going wrong rather than at the answer as a whole. Faithfulness, also called groundedness, is the fraction of the claims in an answer that the retrieved passages support, so it measures whether the model stayed inside the evidence it was handed. Answer relevance is how directly the answer addresses the question that was asked. This chapter is about those two: what each one compares against, and the quadrant that appears once you have both.

Two metrics, two right-hand sides#

A specimen first, because the failure is easier to recognise than to describe.

Q: "What is the refund window on annual plans?"

retrieved[0]  (p-4412, billing/annual-plans.md, score 0.83)
  "Annual plans renew automatically on the anniversary of the original
   purchase date. Renewal notices are sent to the account owner 30 days
   before the renewal date. Changes to seat count take effect at renewal."

answer:
  "Annual plans renew automatically on the anniversary of your original
   purchase date, and we send a renewal notice to the account owner 30 days
   in advance. [p-4412]"

faithfulness: 1.00      answer_relevance: 0.54      user rating: thumbs down

Retrieval did its job: billing/annual-plans.md is the correct page for a question about annual plans, and it was ranked first. Generation did something defensible too, in that every clause in the response appears verbatim or near-verbatim in the passage it cites, and no sentence was invented. What the user asked for was the number of days after purchase during which they could get their money back, and that number lives on a different page, billing/refunds.md, which was not retrieved. The response is a competent summary of a document that does not answer the question.

Metrics of this shape exist because the obvious alternative measures too many things at once. A single answer-quality score, whether it comes from a reviewer's thumbs-up or from a model asked to rate the reply out of five, blends grounding, relevance, tone and completeness into one number that cannot say which component should receive the fix. Splitting the score is topic 3.14's component-versus-end-to-end argument applied to a pipeline whose components are retrieval and generation, and each of the resulting metrics is defined by the two artifacts it compares while staying silent about everything else. You meet them as named functions in the eval libraries teams adopt — Ragas and DeepEval both ship faithfulness, answer relevance and context metrics under roughly these names — as columns on the dashboard a nightly eval run writes to, and as thresholds that fail a build. The rest of the topic adds context precision and context recall on the retrieval side, and citation accuracy on the link between a sentence and the passage it names.

Written out as comparisons, the two definitions have an asymmetry that does most of the work in this topic. Faithfulness takes the answer and the retrieved passages, and asks whether the passages support what the answer asserts; answer relevance takes the answer and the question, and asks whether the answer addresses what was asked. The question does not appear in the faithfulness computation at all, and the retrieved passages do not appear in the relevance computation at all. That is not an implementation detail to be tidied up later; it is the entire content of the distinction, and it survives every reimplementation of both metrics.

Both scores are produced by a judge, which in this topic means a separate model call that reads the texts, applies a rubric and returns a verdict rather than prose. You can see the separation most clearly in the signatures, because each function is missing an argument the other one has.

def faithfulness(answer: str, passages: list[str], judge: Judge) -> float:
    """Fraction of atomic claims in `answer` entailed by `passages`.
    Does not receive the question. Does not receive a reference answer."""

def answer_relevance(answer: str, question: str, embed: Embedder,
                     judge: Judge, n: int = 3) -> float:
    """How directly `answer` addresses `question`.
    Does not receive the retrieved passages."""

The word entailed in the first docstring carries weight that the rest of this topic leans on: a passage entails a claim when somebody reading that passage alone would have to accept the claim, which is a stricter relation than the claim merely sounding consistent with what the passage says.

The Ragas paper, which is the source most teams' definitions trace back to, computes answer relevance in a way that makes its blindness to retrieval concrete. The judge reads only the generated answer and writes the n questions that answer would be a good reply to, and the score is the mean cosine similarity between the embeddings of those reverse-engineered questions and the embedding of the real one.

REVERSE_Q = """Read the answer below. Write {n} distinct questions that this
answer would be a direct and complete reply to. Output one per line.

ANSWER:
{answer}
"""

def answer_relevance(answer, question, embed, judge, n=3):
    generated = judge(REVERSE_Q.format(answer=answer, n=n)).splitlines()
    qv = embed(question)
    return sum(cosine(qv, embed(g)) for g in generated) / len(generated)

Read that as a round trip. If an answer really is a direct reply to the question, then working backwards from the answer to the question it answers should land close to where you started, and the distance travelled is the score. Nothing in the round trip touches a passage, which is why the function takes no passages argument and cannot be made to notice one.

Run it on the specimen above and the judge produces "When do annual plans renew?", "How much notice is given before an annual plan renews?" and "When does a seat count change take effect?" — three reasonable questions, none of which is the one that was asked. Their cosine similarities against "What is the refund window on annual plans?" came out at 0.61, 0.44 and 0.58, and the score is the mean of those three: they add to a little over one and six tenths, and a third of that is 0.54, against a shipping threshold of 0.70. Faithfulness on the same answer is 1.00, since both of its claims are lifted from p-4412.

The three similarities are worth reading individually rather than only as a mean, because their spread says what the metric is sensitive to. All three reverse questions share the vocabulary of annual plans and billing with the real one, which is why none of them lands near zero. The lowest, 0.44, is the seat-count question, which shares the least subject matter; the highest, 0.61, is the renewal-timing question, which shares the most. So the metric is tracking topical overlap and grading the gap between "about the same document" and "about the same question" on a scale where both ends are occupied.

That is why the score is 0.54 rather than 0.05, and it is worth sitting with. Answer relevance implemented through embeddings is a soft measure that degrades gracefully, and topical neighbours score in the middle of the range rather than at the bottom. An answer about a completely unrelated subject would land near 0.15. The band between roughly 0.45 and 0.70 is where "grounded in the right document, addressing the wrong question" lives, which means a threshold set too low turns the metric into a topic-match detector that this failure passes.

The quadrant, and which cell the arguments are in#

With two independent scores there are four combinations, and each one has a different cause and a different owner.

Almost every dispute about a fine-looking answer is a dispute about which of the two off-diagonal cells you are in, and the two have nothing in common operationally. The bottom-left cell is fixed by retrieving different passages, by rewriting the query, or by adding an abstention path, meaning a route that replies that the documentation does not cover the question instead of answering it, when nothing retrieved is on point. The top-right cell is fixed by changing what the generator is allowed to say when the context is thin. Spending a sprint on a reranker to fix a top-right problem produces exactly the change in the numbers you would expect, which is none.

The single score that hides this is the composite "answer quality" that most teams reach for first, because a judge asked to rate an answer from 1 to 5 on helpfulness will average across both dimensions and return something in the middle for both off-diagonal cells. A 0.89 mean on that composite is compatible with a system whose failures are entirely in one cell, and it gives you no way to find out which.

Cross-tabulating the two scores per question is what turns them into a work plan. On the same 400-question eval set that produced the refund ticket, mean faithfulness was 0.93 and mean answer relevance 0.78 — two mildly imperfect numbers — and applying a pass threshold of 0.85 on faithfulness and 0.70 on relevance splits the set into the four cells. Of the 400, 306 passed both, 62 were faithful and irrelevant, 24 were relevant and unfaithful, and 8 failed both. Those four counts account for the whole set with nothing unclassified, which is what makes the shares below shares of something rather than proportions of a sample.

Read the counts against the means and the two quantities come apart in an instructive way. Faithfulness passed on 306 plus 62 questions, which is a shade over ninety-two in a hundred, sitting a point below its mean of 0.93. Relevance passed on 306 plus 24, a little over eighty-two in a hundred, sitting four points above its mean of 0.78. The gap between a pass rate and a mean is a fact about the shape of the score distribution rather than about the pipeline. Faithfulness is a ratio over claims, and most answers have every claim supported or one claim unsupported, so its scores pile up near 1.00 and a threshold at 0.85 separates them cleanly. Answer relevance is the soft embedding measure above, with a long middle band, so a large group of questions sit somewhere in the 0.45-to-0.70 region and drag the mean down without changing how many cross the line. A team that reports means and a team that reports pass rates will disagree about which metric is in worse shape, and both will be reading the same run.

The shape of the distribution is the actionable part. The dominant failure was not the model inventing things, it was the model summarising near-miss passages, so the 62 questions in the bottom-left cell were a retrieval and query-understanding backlog rather than a generation one. Reported as a single 0.85-ish quality average, the same run would have justified a sprint on prompt wording, which could have moved 24 questions at most — fewer than half the number sitting in the other cell.

These two comparisons displaced an older way of scoring the same outputs. Before LLM judges were cheap enough to run per build, RAG answers were scored with ROUGE-L or BERTScore against a human-written gold answer, the first counting the longest word sequences the two texts share and the second comparing their token embeddings. Both require a reference for every question and both reward surface overlap, so a correct answer phrased differently from the gold text scores badly while a fluent paraphrase of the wrong document scores respectably on shared vocabulary. Faithfulness and answer relevance need no gold answer at all — one compares against text your pipeline already produced, the other against text the user already typed — which is what makes them runnable on live traffic where no reference exists.

A judge holding only the question and the answer#

The failure such a setup is structurally blind to is the whole of the faithfulness axis, because the retrieved passages were never in the comparison. Whatever it reports, it is reporting on how well an answer matches a question, and support is not a property of that pair.

Here is a concrete answer it scores highly. Ask "What is the refund window on annual plans?" and have the model reply "Annual plans can be refunded in full within 30 days of the purchase date; after that, a prorated credit is issued for the unused portion of the term." That reply is on topic, specific, confidently phrased, and structured exactly like a real policy answer, so a judge comparing it to the question will score it 0.9 or better. Nothing in the retrieved passages says any of it. The prorated-credit clause in particular is the kind of plausible invention a model produces when it has seen a hundred SaaS refund policies during pretraining, and the eval as configured cannot tell that answer apart from the one where the model read the actual policy page.

The tempting repair is to add a second judge and average the two verdicts, on the reasonable principle that inter-judge agreement is what makes an LLM verdict trustworthy. That principle is sound and it is attached to the wrong problem here. Work through what the second judge is given: the question and the answer, the same two texts as the first. The fabricated reply genuinely is a direct and well-formed answer to the question, so both judges will call it relevant, and they will agree with each other for the same reason either of them was right on its own terms. Their agreement is therefore constant in the property that matters — it reads the same whether the passages supported the prorated credit or never mentioned it — because neither judge was shown a passage. Adding a second opinion buys precision on a measurement that was never looking at the evidence, and precision on the wrong quantity leaves the failure exactly where it was.

The addition that helps is a second metric with a different right-hand side rather than a second opinion on the same one. Faithfulness compares the claims in the answer against the retrieved passages, and it is what catches the invented prorated credit. Answer relevance compares the answer against the question, and it is what catches the renewal-anniversary summary. Both were needed on the two examples in this chapter, and each one passes the case the other fails.

Correctness is a third comparison#

A faithful answer can be wrong, because the passage it faithfully reproduces can be stale, can describe a different plan tier, or can be a customer-facing draft that was never approved. A relevant answer can be wrong for the ordinary reason that it is confidently mistaken. The pair together does not add up to correctness either: an answer that is faithful to a retrieved passage and directly addresses the question is wrong whenever the corpus is wrong, and no metric computed from the pipeline's own inputs can detect that. Every input those two metrics receive — the question, the answer, the passages — is something the system already had, and none of them is a record of what is actually true. Judging correctness requires a reference answer written by somebody who knows the true state of the world, which is a different comparison against a different right-hand side and is scored end to end rather than per component, in the sense topic 3.14 set out.

There is also a case where faithfulness is the wrong instrument entirely. If your system is allowed to answer from the model's own knowledge when retrieval returns nothing useful — a reasonable design for a general assistant with a small corpus — then a correct answer produced with an empty context scores 0.0 on faithfulness while helping the user. The metric is defined for pipelines that promise every claim comes from the corpus, and applying it to one that makes no such promise produces a number that punishes the intended behaviour. That is a scoping condition rather than a defect in the metric, and the way to state it is that faithfulness measures adherence to a promise your architecture may or may not be making.

Two references, then, and two scores that cannot substitute for each other. Faithfulness holds an answer against the passages that were retrieved and reports whether the model stayed inside them, without any opinion about whether those passages were the right ones. Answer relevance holds the answer against the question and reports whether it addressed what was asked, without any opinion about where its content came from. The quadrant they define is what turns a complaint about a fine-looking answer into a routed piece of work, and the cross-tabulation of the two is what tells you which cell your failures are actually in rather than which one is easiest to imagine.

Relevance failures like the refund summary have one convenient property: a person who reads the question and the answer side by side notices them, because the mismatch is right there in the text. Faithfulness failures do not work that way. The fabricated prorated-credit sentence reads exactly like the real thing, and a reviewer skimming it has no way to tell without going back to the passages, which is why a review process that feels thorough can pass a batch of answers that an audit later finds unsupported.