Type-aware equality

String normalization runs out the moment a field is not consumed as a string. "2024-03-01", "March 1, 2024" and "1 March 2024" are one value to a job that parses a date and charges from it, and no sequence of lowercasing, trimming or stripping will unify them; meanwhile "March 2, 2024" differs from one of them by a single character and must fail. The strings that have to merge share almost nothing, and the strings that have to stay apart look nearly identical.

That inversion is not bad luck about these particular strings. It follows from what a written date is. A date is a value with three components — a year, a month and a day — and writing one down means choosing an order for those components, a separator to put between them, and whether the month appears as a numeral or as a name. Two writers who choose differently produce strings that may share nothing beyond the digits of the year, while two writers who choose identically and disagree about the day produce strings differing in one character. The distance between two written dates is therefore mostly a measure of the encoding choices, and the quantity the eval wants is the value underneath them. That also disposes of the obvious next move, which is to make the string rules more aggressive. A rule aggressive enough to pull March 1, 2024 onto 2024-03-01 has to discard almost everything that distinguishes one written date from another, and on the way it will merge March 1 with March 2. Any comparison performed on the strings is measuring the wrong thing however carefully it is tuned, which is why the repair has to be to decode first rather than to compare better.

Type-aware equality is the repair: a field with a type is compared after coercion to that type, and the coercion used is the one the consumer of the field performs. The interesting part of the design is not how to write the comparison — it is two lines — but deciding what the coercion is, which turns out to be a question about the running system rather than about the eval. The same move works on money, quantities, booleans and opaque identifiers, and it has exactly two failure modes, both of which follow from importing someone else's parser rather than writing your own. A parser lenient enough to accept the model's worst outputs will pass answers that should have failed, because leniency was a virtue where it came from and is a defect here. And a field with more than one consumer has no single coercion to import, which means the question the eval was supposed to settle has to be settled somewhere else first.

What a date field's failures actually look like#

A contract extractor scoring 62% on effective_date across 240 documents produced 91 failures that are almost entirely shapes rather than dates. The reference set had been transcribed from the contracts themselves, so it carried whatever wording each contract used; the model normalised toward ISO 8601 — the international standard shape for a date, year then month then day, 2024-03-01 — about two-thirds of the time and echoed the document's own wording the rest.

    prediction            reference
  1 2024-03-01            March 1, 2024
  2 March 1, 2024         1 March 2024
  3 01 March 2024         2024-03-01
  4 2024-03-01T00:00:00Z  2024-03-01
  5 March 2, 2024         March 1, 2024

Only the last row is a defect, and it is the one a billing job would care about: this field is read by a nightly job that computes the first billing period from the effective date, so a day of drift is a day of charges. Everything above it is the same instant written four ways.

A list of accepted formats is a list that grows#

The thing people build next is a format table: parse each side against a list of known patterns, take the first that matches, compare the results.

from datetime import datetime

FORMATS = ["%Y-%m-%d", "%B %d, %Y", "%d %B %Y", "%m/%d/%Y"]

def parse_known(s: str) -> datetime | None:
    for fmt in FORMATS:
        try:
            return datetime.strptime(s.strip(), fmt)
        except ValueError:
            continue
    return None

This works on the day it is written and then decays, because the list is a guess about what documents contain. Mar. 1, 2024, 1st March 2024, 2024/03/01 and a European supplier's 01.03.2024 all arrive eventually, and each one arrives as a bug report against the eval rather than against the model. Every addition is a judgement call made by whoever is on the eval that week, and none of those judgements is connected to anything the system does with the value.

The connection is available and mostly goes unused. A billing job that consumes effective_date already contains a function that turns the string into a date, because it cannot charge from a string. Whatever that function accepts is what production accepts; whatever it rejects lands in the same error queue a wrong extraction would. Copying it into the eval is not a convenience — it makes the score a statement about the running system.

The comparator is the consumer's parser#

from billing.ingest import parse_effective_date   # the job's own function

def score_effective_date(prediction: str, reference: str) -> bool:
    expected = parse_effective_date(reference)   # raises on a bad fixture
    try:
        actual = parse_effective_date(prediction)
    except ValueError:
        return False                             # model emitted something unusable
    return actual == expected

Nine lines and no format list. The two sides are not symmetric, and the structure is worth reading off explicitly, because the asymmetry is where suites quietly break:

The two calls to parse_effective_date in that function look identical and mean opposite things, and the reason is that the two strings have different authors. A reference is a human assertion about what the correct value is; a prediction is a machine's attempt to produce it. An assertion that cannot be decoded is not a wrong answer, because it was never an answer at all — it is a broken premise, and there is nothing left for a prediction to be compared against. An attempt that cannot be decoded is exactly a wrong answer, because production would reject it too. One exception type, raised about two strings with different provenance, therefore has to be routed two different ways.

An unparseable prediction is a model failure and should score zero. An unparseable reference is a defect in the eval set — in the fixture, meaning the stored example and the hand-written reference value the suite asserts against — and if you catch both with the same try you will silently convert broken fixtures into passes or failures depending on which branch you wrote. Letting the reference parse raise is the cheap way to keep the fixtures honest; a suite that scores 240 rows should be loud when one of its 240 truths is not a date.

What the pattern cannot survive is a comparator that turns failures into equalities. The version everyone writes first does exactly that:

def parse_or_none(s):
    try:
        return parse_effective_date(s)
    except ValueError:
        return None

# prediction "N/A", reference "not stated"  →  None == None  →  pass

Two unparseable strings compare equal, so the worst rows in the set score as the best ones. Any comparator that maps failure onto a sentinel needs the sentinel excluded from equality, and it is simpler not to have one.

The mechanism underneath that is worth naming, because it comes back later in this chapter wearing a different surface. parse_or_none is a total function: it returns a value for every input it is given, including the inputs it cannot understand, and the value it returns for all of those is the same one. Every distinct failure is therefore mapped onto a single point in the value domain, and equality on that domain reports two failures as a match — correctly, because after the coercion they genuinely are the same point. The defect is not in None specifically, since any sentinel behaves this way. It is in choosing a coercion whose range contains a value meaning this could not be done. A coercion that raises has no such value in its range, which is why the version that raises needs no special case to stay honest.

The same move on money, quantities and identifiers#

Nothing in this is specific to dates. A field with a type gets compared after coercion, and the table below is the same decision made five times.

The last row matters as much as the others. When the consumer does not interpret the string, there is no coercion to copy and the comparison stays exact, however arbitrary the difference looks. A purchase order number that differs by a leading zero is a different key in the ERP — the enterprise resource planning system, the finance package the invoices are ultimately posted into — and an eval that forgives it is describing a system that does not exist.

The net_weight row is where the pattern needs care rather than mechanics. Once you convert units you are comparing floats, and exact equality on floats is the wrong test — but the tolerance is not yours to pick. If the warehouse system rounds to the nearest gram before storing, then agreement to the nearest gram is the equality relation and abs(a - b) < 0.5 is the comparator; if it rounds to the nearest kilogram, a two-gram difference is not merely acceptable, it is invisible downstream.

abs(a - b) < 0.5 is a proxy for the relation that actually holds downstream, and it is worth knowing that it is a proxy rather than a restatement. What the warehouse makes true is that two weights are the same value when they round to the same integer gram. Two weights sitting a little either side of a rounding boundary differ by less than half a gram and still store as different numbers, so the tolerance accepts a pair the warehouse would separate. The gap is narrow and usually not worth closing. The reason to notice it at all is that a tolerance is an equality relation in its own right, written by hand, and therefore answerable to the same question as every other step in this topic. A tolerance chosen because it makes the number look better is the same failure as the previous chapter's token sort wearing a numeric costume.

Fields that look like they have no type often have the most brittle one. auto_renew arrives from contracts as Yes, Y, true, TRUE, [x] and, on one template, the word Annual, and the consumer coerces it to a boolean before deciding whether to raise an invoice next year. Comparing the raw strings fails constantly; comparing bool(value) in Python is worse, because bool("false") is True and every rendering, including the negative ones, collapses to a single value that always matches. The coercion has to be the consumer's explicit mapping — an allow-list from token to boolean, with anything unmapped raising — and the eval imports that mapping rather than approximating it. The general hazard is the same one the parse_or_none sentinel had: any coercion that turns unrecognised input into a legal value converts the model's worst outputs into passes.

bool("false") being True is not a quirk of that particular word. In Python, bool applied to a string asks only whether the string is non-empty and never looks at what the string says, so Yes, Y, true, TRUE, [x] and Annual all come back True — and so does every non-empty rendering of the opposite. The field then scores close to perfectly while carrying no information whatsoever, because prediction and reference agree on every row for the same reason: the coercion discarded the one bit that distinguished them. That is the range collapsing again, exactly as it did onto None, except that this time it collapses onto the value meaning yes, which is why the symptom is a suspiciously good number rather than a suspiciously bad one.

Where a lenient parser passes a wrong answer#

Copying the consumer's parser means inheriting the consumer's leniency, and general-purpose date parsers are extremely lenient. If parse_effective_date is a thin wrapper over dateutil, two behaviours will surprise you.

from dateutil import parser as dtp

dtp.parse("March 2024").date()
# datetime.date(2024, 3, 8)   ← day filled in from today's date

dtp.parse("01/03/2024").date()
# datetime.date(2024, 1, 3)   ← month-first, silently

The first is the dangerous one. A model that returns "March 2024" has not extracted a date, and dateutil will hand back a complete date anyway, filling the missing day from the clock — so the same prediction passes on the first of the month and fails on the eighth. The second is dangerous in a quieter way: a European contract reading 01/03/2024 means 1 March, the parser reads 3 January, and the eval and the billing job agree with each other while both being wrong about the document.

The day-fill is worth one further step, because of what it does to the property the previous chapter leaned on. A parser of this kind promises to return a complete date, and "March 2024" does not contain one, so the missing component has to come from outside the string — and the default source is today. That makes the comparator a function of three things rather than two: the prediction, the reference, and the day the suite ran. A scorer that reads the clock is not deterministic, and determinism is what the previous chapter leaned on to conclude that the definition of equality was the only thing in a comparator worth auditing. A suite that has lost it can move without the model, the prompt or the fixtures having changed at all. The output printed above was produced on the eighth, which is why "March 2024" decoded to the eighth; against a reference of 2024-03-01 that row fails. Run the identical suite over the identical fixtures on the first of the month and it passes.

Neither of these is a reason to abandon the approach, because both are already happening in production. That is the point of using the consumer's parser: the eval now reproduces the system's real behaviour instead of an idealised version of it, and both surprises become tractable bugs with owners. The day-fill gets a strictness flag, the ambiguity gets a locale from the contract's jurisdiction, and the eval will show the change as movement in the number the moment the parser ships.

The belief to be careful with here is that importing the consumer's parser makes the eval correct. It does not. It makes the eval faithful, which is a different property and a weaker one: a faithful eval reports what the running system will do with the model's output. Faithful and correct coincide only where the running system is right, and the 01/03/2024 case is exactly where they come apart. The reference set was transcribed verbatim from the contracts, so where a contract wrote 01/03/2024 the reference carries that shape too; both sides then decode to 3 January, the row matches, and the suite records a pass for a date the contract gave as 1 March. That error is invisible to the score by construction, because the score is computed with the function that causes it, and no amount of re-running will surface it. Catching it takes a different instrument altogether — the parser's own tests, written against what the documents mean rather than against what the pipeline currently does.

The same class of mistake hides in the type you compare. If the parser returns a datetime rather than a date and the job stores it in a local timezone, the comparison inherits an offset:

from datetime import datetime
from zoneinfo import ZoneInfo

datetime.fromisoformat("2024-03-01T00:00:00+00:00") \
        .astimezone(ZoneInfo("America/Los_Angeles")).date()
# datetime.date(2024, 2, 29)

The model returned the right instant and the comparator reports the wrong day, because midnight UTC is the previous afternoon in California. Two steps produce that output and both are worth following. The instant named is the very start of 1 March in UTC; Los Angeles runs behind UTC, so the same moment falls on the previous calendar day there, and .date() then throws the time away and keeps whichever day the conversion happened to land on. The previous day is 29 February rather than 28 because 2024 is a leap year, which is the detail that makes the printed output look like a typo when it is not. Compare at the granularity the consumer bills at — a date, not an instant — and the problem does not arise.

A field can also be scored before its consumer exists, which happens constantly on new pipelines and looks at first like a hole in the whole approach. It is not, provided the order of work is right. If nothing downstream parses renewal_term yet, the thing to write is not a comparator but the coercion the future consumer will use — the accepted shapes, the rejected ones, the unit — and then the comparator on top of it. That artefact is useful to the team building the consumer, it is reviewable by whoever owns the contract, and it prevents the most common version of this mistake, which is a scorer whose notion of equality was invented by whoever happened to be writing the eval that week and then inherited by the production code six weeks later.

When two consumers disagree#

The pattern assumes there is a consumer. Fields commonly have several, and they do not always coerce alike. In one contract pipeline effective_date is read by the billing job, and it is also loaded into an analytics warehouse whose ingestion casts the column with a stricter function that accepts ISO 8601 and nothing else. March 1, 2024 is a perfectly good value for one consumer and a null for the other.

Averaging over that disagreement produces a number that describes neither system. There are two defensible resolutions and the choice is a product decision rather than a technical one. Score against the strictest consumer, which reports the fraction of outputs usable everywhere and will read lower than any single consumer's experience; or score the field once per consumer and report both, which is more honest and costs a column on the dashboard. What is not defensible is picking the lenient parser because it produces the better number, which is the same failure as picking a lenient normalizer, with an import statement in front of it.

The disagreement is also information worth acting on rather than scoring around. Two coercions of the same field in one company is usually an accident of history, and the eval is often the first place it becomes visible, because nothing else in the system compares the two. Reporting effective_date at 91% against the billing parser and 64% against the warehouse's is a bug report about the pipeline, not about the model — and the fix is one shared parser, after which the second column can go away.

The warehouse's figure is not an independent fact about the model either, and seeing why makes it more useful while it is still on the dashboard. That ingestion accepts ISO 8601 and rejects everything else, so a prediction can only score against it if the model happened to write the date in ISO — which it does about two-thirds of the time. The warehouse column is capped by that share before correctness enters into it at all, and the 64% sits just under the cap, the gap being the ISO-shaped predictions that name the wrong day. Read that way, the second column is only partly measuring extraction quality; most of what it measures is output shape, which is a fact about the prompt rather than about whether the model read the contract correctly.

What type-aware equality buys, then, is a score that moves when the system's behaviour moves and stays still otherwise. The format list could not do that, because it encoded one engineer's guess about documents; the parser encodes what production will accept, which is the only definition of a usable value that anybody downstream is bound by. The cost is a dependency, and it is a real one: the eval is now exactly as correct as the function it imports, which is why that function gets rejection tests of its own and why those tests are not part of the suite but the thing that makes the suite mean anything.

Type-aware equality covers every field that has a type, which in an extraction pipeline is most of them. It runs out completely the moment the output has no type at all. A summary, an explanation, a customer-facing answer — there is no parser downstream to appeal to, no canonical form to compare, and the only thing on hand is a human-written reference and the model's text. That is the case the next chapter takes up, where the standard move is to count the words the two share.