Reference-guided grading

Reference-guided grading shows the judge a known-good answer alongside the candidate and asks it to rule on the candidate against that reference, rather than against the question alone. It exists because a question in English usually underspecifies its own answer: "how many orders did each enterprise customer place in Q3" does not say which timestamp column defines "in Q3", does not say whether the quarter boundary is inclusive, and does not say whether customers with no orders should appear. A reference settles all three without anybody having to write them down.

It also settles a great many things nobody asked it to settle, and that is the subject of this chapter. The artefact that fixes the semantics is an implementation, and every incidental property of that implementation arrives in the judge's context wearing the same authority as the parts that matter. What follows is how to say in the rubric which is which, how to hand the comparison to a database rather than to a model, and why the false fails this produces are still preferable to the alternative.

One artefact, two jobs#

Take a text-to-SQL suite. The question is "how many orders did each enterprise customer place in Q3?" and an analyst's reference query joins two tables:

-- reference
SELECT c.customer_id, c.name, COUNT(o.order_id) AS order_count
FROM customers c
JOIN orders o ON o.customer_id = c.customer_id
WHERE c.plan = 'enterprise'
  AND o.placed_at >= '2026-07-01'
  AND o.placed_at <  '2026-10-01'
GROUP BY c.customer_id, c.name;

The model produced a correlated subquery instead:

-- candidate
SELECT c.customer_id, c.name,
       (SELECT COUNT(*)
        FROM orders o
        WHERE o.customer_id = c.customer_id
          AND o.placed_at >= '2026-07-01'
          AND o.placed_at <  '2026-10-01') AS order_count
FROM customers c
WHERE c.plan = 'enterprise';

The difference between those two queries is not stylistic, and locating it exactly is what makes the rest of this chapter checkable. An inner join emits a row only where a match exists on both sides. A customer with no order inside the window has nothing to pair with, so no row for that customer survives the join, and the customer is therefore absent from the grouped output altogether rather than present with a count of nothing. The correlated subquery works the other way round. The outer query selects enterprise customers first, and the inner query runs once for each customer that has already survived that filter, so a customer with no matching orders still produces an outer row and the count over an empty set comes back as zero.

The two do not return identical rows in general — the subquery version keeps enterprise customers with zero Q3 orders, which the inner join drops — but on the seeded database this suite runs against, every enterprise customer had at least one order, so the outputs matched row for row. The judge failed it:

{
  "verdict": "fail",
  "reason": "The candidate does not follow the reference approach. The reference
             joins customers to orders and aggregates with COUNT(o.order_id),
             while the candidate uses a correlated subquery. The structure
             differs materially from the expected solution."
}

Every clause of that explanation is factually accurate. The verdict is still wrong, and the reason is that the reference encodes two different kinds of information the judge has no way to separate.

The first is what a correct answer must produce: one row per enterprise customer, with a count of their orders in the July-to-September window, using the placed_at column and a half-open interval, one whose start is included and whose end is excluded. That is the semantic target, it is not stated anywhere else, and it is the reason a reference is worth having.

The second is one particular way of computing it: a join rather than a subquery, COUNT(o.order_id) rather than COUNT(*), this grouping, this column order. None of that is required by the question. It is an implementation the analyst happened to write. Given a single artefact carrying both, the judge treats every property of it as normative, because nothing in the prompt tells it which properties are load-bearing, and it cannot infer that from the SQL — a join and a correlated subquery are visibly different objects and the reference is visibly one of them. The failure is not the judge being careless; it is the judge doing exactly what it was asked, with a rubric that never said what a reference is for.

The layer under that is worth going down to, because it decides what any repair can achieve. A judge is a language model, which is to say a function that takes a piece of text and scores what could come next; a verdict is a continuation of a prompt, and that prompt is one flat string. There is no reference channel. The rubric, the question, the candidate query and the reference query are concatenated into that string, and the verdict is produced by conditioning on all of it at once. Nothing in the arrangement distinguishes the characters that arrived from the reference from the characters that arrived from the rubric, and nothing marks a stretch of the reference as advisory rather than binding.

Both halves of reference-guided grading follow from that one property. The judge can check a candidate's facts against the reference — the window, the column, the grain — precisely because the reference is in front of it while the verdict is being scored, and that is the entire reason the technique works. The reference's join strategy is in front of it on identical terms, so it conditions the verdict on identical terms. The benefit and the defect are one mechanism seen from two sides, which is why instructing the judge to ignore the implementation does not remove the second: the instruction is more text in the same string, competing with the artefact rather than filtering it.

The sentence a reader is likely to write after meeting this technique is that supplying a reference makes the grade objective, and it is cheaper to block that here than after a suite has been built on it. The reading is tempting for a real reason. A reference is a fixed artefact, checked into a repository and reviewable by anyone, and comparing something against a fixed artefact is what a test does. What actually changed is narrower than that. The reference fixes what the judge is conditioned on; it does not change how the verdict is produced, which is still one model continuing one string. The open question has moved rather than closed — from "does this query answer the question", which the judge had to interpret, to "does this query match the reference", which it also has to interpret, now over a longer text with more in it to disagree about. Since the judge cannot be given a filter, the only place the distinction between the two jobs can be written down is the rubric.

Saying it in the rubric#

The repair is a clause that names the dimensions along which divergence is permitted, specifically enough that a grader could apply it:

reference_semantics: |
  The reference query is one correct implementation. It defines WHAT a correct
  answer must return, not HOW it must be computed.

  PASS a candidate that differs from the reference in any of:
    - join strategy (inner join, left join with filter, correlated subquery,
      lateral join, CTE) when the returned rows are the same
    - COUNT(*) vs COUNT(column) when the column is non-nullable in scope
    - column aliases, column order, row order when the question does not
      specify an ordering
    - formatting, casing, whitespace, comments

  FAIL a candidate that differs from the reference in any of:
    - the set of rows returned, including rows present in one and absent
      from the other
    - the grain of aggregation (per customer vs per customer per month)
    - the filter window, including boundary inclusivity
    - the columns projected, when the question asked for them

  If a difference in approach CHANGES which rows are returned — for example a
  subquery that retains customers with zero matching orders where the
  reference's inner join drops them — that is a row-set difference and is
  graded as one. Say which rows differ.

That last paragraph is the one that saves the clause from being a licence to pass anything. The subquery above genuinely does behave differently on data where an enterprise customer has no Q3 orders; it matched only because the seeded data did not contain that case. A rubric that says "different approaches pass" without tying the exemption to the row set turns a real semantic difference into a permitted stylistic one, and the judge — which has no data to run against — will happily wave through a left-join-versus-inner-join swap that changes the answer on production data.

One line of the permitted list carries a condition that is easy to read past, and working it out shows how narrow the permissions have to be. COUNT(*) counts rows; COUNT(column) counts only the rows where that column is not null. Inside the reference's inner join the two expressions can never disagree, because a row exists there only when an order matched, and a matched order has an order_id. Move the same pair into a left join and they come apart. An enterprise customer with no Q3 orders survives that join with the whole orders side filled in as nulls, so COUNT(*) sees one row and returns one, while COUNT(o.order_id) sees one null and returns zero. "Non-nullable in scope" is therefore doing work rather than hedging: the scope is the query the expression ends up in, not the table definition, and the same column is nullable in one query and not in the other.

That condition is what ties this clause to the artefact further down this chapter, whose clarification requires enterprise customers with no Q3 orders to appear with a count of zero and whose accepted reference is accordingly written as a left join. Only one of the two expressions produces that row. A rubric that had granted the COUNT exemption unconditionally would have licensed the judge to accept a query returning one where the answer is zero, which is a value error dressed as a formatting preference — the same shape of mistake as the join permission, and avoided by the same discipline of tying every exemption to what comes back.

Execute what can be executed#

Which points at the better arrangement wherever a warehouse is available to run against. The row-set comparison the rubric keeps talking about is a computation, not a judgment, and asking a model to simulate it from two SQL strings is asking for an error the model cannot avoid making.

Why a database can be trusted with that comparison where a model cannot is worth stating rather than assuming, because it is the property that decides which parts of grading can leave the judge at all. Row-set equality is settled entirely by the data: given one fixture and two queries, the answer is fixed before anybody asks, the procedure that reads it off is the same procedure every time, and it depends on nothing about how either query was phrased. A judge handed the same question is doing something structurally different — scoring a continuation of a prompt that happens to contain two SQL strings — so its answer can move with the wording, with which query appeared first, and with how familiar each formulation looks. The first procedure has an answer anyone can recompute; the second has an opinion that has to be calibrated against people before it means anything. Every check a suite moves from the second kind to the first is a check it no longer has to validate, which is a larger saving than the accuracy difference alone suggests.

def grade(question: str, candidate: str, reference: str, fixtures: list[str]):
    if not is_read_only(candidate):          # reject DDL/DML outright
        return {"verdict": "fail", "why": "not a read-only query"}
    try:
        cand_rows = [run(candidate, f) for f in fixtures]
    except SQLError as e:
        return {"verdict": "fail", "why": f"does not execute: {e}"}

    ref_rows = [run(reference, f) for f in fixtures]
    if all(same_rows(c, r) for c, r in zip(cand_rows, ref_rows)):
        return {"verdict": "pass", "why": "result set matches on all fixtures"}

    # Rows differ: the judge explains and classifies, it does not decide equality.
    return judge_row_difference(question, candidate, reference,
                                cand_rows, ref_rows)

Two things make this materially better than judging the text. Equality is decided by execution rather than inference, so a correlated subquery that returns the reference's rows passes without any rubric language about join strategies. And the fixture set — the small seeded databases every query is executed against, each one built to contain a particular case — is where the earlier trap gets closed: adding a fixture in which one enterprise customer has zero Q3 orders makes the inner-join-versus-subquery difference visible as a row-set difference, on that fixture, automatically. The judge's remaining job is the part execution cannot do — saying why two row sets differ, in terms a person can act on, which is the difference between "17 failures" and "17 failures, 11 of them a grain error on monthly aggregates".

A reference is a set, not a query#

Once the reference's job is stated as "define what a correct answer returns", the single-query format starts looking like an accident of how these suites get built. Some questions genuinely admit more than one correct row set, and the honest artefact says so:

- id: q-0142
  question: "How many orders did each enterprise customer place in Q3?"
  clarification: >
    Q3 is 2026-07-01 inclusive to 2026-10-01 exclusive, on orders.placed_at.
    Enterprise customers with zero Q3 orders MUST appear with a count of 0.
  accepted_references:
    - name: left_join
      sql: |
        SELECT c.customer_id, c.name, COUNT(o.order_id) AS order_count
        FROM customers c
        LEFT JOIN orders o
          ON o.customer_id = c.customer_id
         AND o.placed_at >= '2026-07-01'
         AND o.placed_at <  '2026-10-01'
        WHERE c.plan = 'enterprise'
        GROUP BY c.customer_id, c.name;
  fixtures: [base, enterprise_with_no_orders, boundary_timestamps]

Three things moved out of the SQL and into their own fields. The clarification states the semantics in English, so a human reviewing a disputed grade can see the intent without reverse-engineering it from a query. The reference list is plural even when it has one entry, which stops anyone from reading "the reference" as "the required approach". And the fixture names pin the cases that distinguish near-equivalent queries.

For items where the correct answer is a property rather than a specific row set, a predicate beats a reference query outright:

def check_q0142(rows) -> bool:
    by_id = {r["customer_id"]: r["order_count"] for r in rows}
    return by_id == {"c_101": 3, "c_104": 0, "c_119": 7}   # order irrelevant

This is more precise than any reference query, because it says exactly what matters and nothing else — not the column order, not the aliases, not whether the result is sorted. It is also more work to write, and it does not double as documentation the way a reference query does.

Where that indifference comes from is visible in the first line. Building a dictionary keyed by customer_id throws the row order away before any comparison happens, since a dictionary has no notion of which entry arrived first, and comparing two dictionaries asks only whether they hold the same keys with the same values. The aliases go the same way: the predicate names the two fields it cares about and ignores everything else the query projected. The price of that precision is coupling. The expected counts are the seeded fixture's counts written into the check, so the fixture and the check have to move together, and c_104 sitting there with a count of 0 is the clarification's requirement that customers with no orders still appear, restated in a form a test runner can enforce. The same construction is also weak in one place worth knowing about: if a candidate returns several rows per customer, the comprehension keeps only the last of them, so a grain error reaches the comparison already flattened and is caught by its values rather than by its shape.

Once execution has decided equality, the judge's remaining job is classification, and the taxonomy is small enough to enumerate in the schema:

{
  "cause": "grain",
  "detail": "Candidate returns 3 rows per customer (one per month of Q3);
             the reference returns 1 row per customer. Candidate groups by
             c.customer_id, DATE_TRUNC('month', o.placed_at).",
  "candidate_rows": 27,
  "reference_rows": 9
}

The permitted values are missing_rows, extra_rows, same_keys_different_values, grain, filter_window, projection, and empty_result. Constraining the judge to that enum rather than letting it write prose is what makes this suite's 160 failures aggregable into strata by cause — the grouping a false-fail audit later samples within, and the only way a correction estimate can be localised to one broken rubric clause. It also removes the judge's opportunity to invent a cause like "does not follow the reference approach" — a category that no longer exists in the output space. Two of those values carry specific diagnostic weight. empty_result on the candidate almost always means a filter that is too tight or a join against the wrong key, and it is worth alerting on separately because an empty result is the failure a model is most likely to produce confidently. same_keys_different_values means the query found the right entities and counted them wrong, which is the shape of an aggregation defect and the single most expensive error class to ship, because the output looks entirely reasonable.

"Aggregable" is carrying weight in that paragraph and is worth unpacking, since it is what connects a schema decision to an audit that has not happened yet. A cause written as free prose comes out as a different string almost every time, even for one defect: two verdicts describing the same grain error will differ in wording, in which columns they name, and in how much of the query they quote. Strings that differ cannot be counted together, so a failure list of that kind supports no grouping at all, and every failure is a category of one. An enum makes the cause a categorical value drawn from a fixed and short set, so the failures partition into groups whose sizes are known exactly. A group whose size is known is precisely what a sample can be allocated across, and allocating a sample across groups is what lets an audit report which rubric clause is generating the failures rather than only how many there are.

The trade you make by dropping the reference#

The tempting conclusion from a false fail is that the reference is over-constraining and should go, leaving a judge that reads the question and the candidate query and rules on whether the query answers it. That removes the anchor doing the useful work, and the errors it stops catching are the expensive ones.

The asymmetry in the two middle rows is the whole argument. A false fail announces itself: the item appears in the failure list, someone reads it, and the rate is estimable from a sample of the failures. A false pass is silent by construction — nobody audits the passes, because the suite says they are fine — so a reference-free judge's error rate is not merely higher, it is unobservable from inside the harness. Trading a measurable error for an unmeasurable one is a bad trade even when the unmeasurable one is smaller, and here it is larger.

"Not countable" in that row needs narrowing before anybody quotes it, because the barrier is not permission. Passes can be audited, and the next chapter audits them, drawing a sample of the passed items and reading them against the row sets exactly as it reads the failed ones. What is missing is any marking of which passes to read. The failure list is short and every item on it is already flagged as suspect, so a fixed review budget spent there meets real errors at the rate they occur among failures, which is high by construction. The same budget spent on the passes meets them at the rate they occur among passes, which is low, and most of the reading comes back empty. Both estimates can be had. One can be had in an afternoon and the other cannot, and a team that only ever has an afternoon will only ever produce the flattering half of the picture.

The specific misses in the table's lower rows share a property. A filter placed in the WHERE clause instead of the ON clause of a left join silently converts it to an inner join. A range written placed_at BETWEEN '2026-07-01' AND '2026-09-30' drops the last day when the column is a timestamp. An aggregate grouped by customer and month when the question asked per customer returns three times too many rows. All three are syntactically clean, idiomatic, well-formatted SQL that a reference-free judge reads as a competent answer, because it is one — it is just answering a slightly different question, and there is nothing in the text to compare against.

Each of those three deserves its steps, because a reader who cannot reconstruct them cannot tell which of their own queries is in danger. A left join emits every row of its left-hand table, filling the right-hand columns with nulls wherever nothing matched. WHERE is applied after that has already happened. A predicate written there against a right-hand column — a date range on o.placed_at, say — is not true of a row whose right-hand side is null, so every unmatched customer is removed again, and what survives is exactly the set an inner join would have produced. The same predicate written into the ON clause takes part in the matching instead, and an unmatched customer still comes through carrying nulls. That is why the accepted reference above puts both date conditions in ON and keeps only c.plan = 'enterprise' — a condition on the left-hand table, which has no null rows to destroy — in WHERE.

BETWEEN goes wrong for a different reason and leaves nothing in the text to see. It is inclusive at both ends, and a bare date literal compared against a timestamp column is read as the first instant of that day. An order placed at any point after midnight on the closing date is therefore greater than the upper bound and is dropped, so the range quietly loses almost the whole of its last day. Writing the interval half-open avoids this outright: >= against the first instant of the quarter and < against the first instant of the next one admits every instant inside the quarter and no instant outside it, with no boundary case to get wrong. That is also why boundary inclusivity sits in the rubric's FAIL list rather than its PASS list. It changes which rows come back, so it is a semantic property of the answer rather than a matter of style.

The grain miss is the one whose arithmetic is already on the page. Grouping by customer and by month across a three-month quarter yields one row per customer per month where the question asked for one row per customer, which is the 27 candidate rows against the reference's 9 in the classification example earlier — three months of rows for each of the nine customers. Nothing in the SQL announces that. Both queries aggregate, both group, both filter to the same window, and the only visible difference is an extra expression in a GROUP BY.

There are cases where the reference stays despite its false fails and despite execution being unavailable: no read access to representative data from the eval environment, non-deterministic sources where two runs of the same query disagree, or semantics the question genuinely underspecifies and the business has settled elsewhere — where "active customer" means something specific that only the analyst's query encodes. In those, the reference is the only place the intended answer exists, and the rubric clause above is the whole of the mitigation.

A reference, then, is worth keeping and worth demoting. It is the only artefact that says what a correct answer returns when the question does not, and it is not an instruction about how to compute it — which has to be written down, because the judge will otherwise assume the opposite. Where a database can be reached, the demotion is total: execution decides equality and the model is left with the job it is actually good at, which is saying in words why two row sets differ.

That leaves the false fails still in the numbers. A suite in this state reports 68% pass, and eleven false fails found by hand in a sample of 50 failures say that 68% is too low. How much too low is a number that can be computed rather than guessed at, and it comes with an uncertainty that has to be reported alongside it.