Python dependency lockfiles

Almost every retrieval system ends in a small piece of code that nobody draws on the architecture diagram. A query arrives, a first pass returns twenty or thirty candidate passages, and something has to decide which few of them are worth the tokens they will cost inside a prompt. Call that piece the reranker: a Python service that takes a query and a list of passages, asks a model to score each passage against the query, and hands back the passages in the order those scores imply.

The reranker runs into four moments where ordinary Python habits stop being enough: setting up the project it lives in, writing the function that scores a passage, evaluating how well it scores, and reading the traceback when it breaks. Four moments, but one piece of code rather than four unrelated examples. Its install, its function signature, its evaluation and its failures each have a way of quietly stopping being reproducible, and each of those stops happens in a different way, at a different moment, in front of a different person.

A pinned file that still isn't reproducible#

The reranker's dependencies live in a file somebody wrote by hand, and every line in it carries an exact version.

anthropic==0.39.0
tokenizers==0.20.3
pydantic==2.9.2
numpy==2.1.3
fastapi==0.115.4

Five names, five exact versions, nothing left to anybody's judgement. In practice, though, that is not enough. A teammate clones the repository four months later, makes a fresh environment, runs pip install -r requirements.txt and starts the service. It raises on the first request it handles: the model client's constructor rejects a keyword argument the reranker has passed since the day it was written.

No line in that file changed in the intervening months, and the teammate's install honoured every one of them. The five names on disk carry the five versions written down. What moved sits underneath: anthropic depends on an HTTP library nobody typed into any file, that library published a release which removed the keyword, and the teammate's install resolved it fresh against whatever was newest on the day it ran.

Hold onto the exact shape of that outcome, because the shape is what makes it hard to catch. Two installs from one file, identical on paper, line for line, with different code on disk and different behaviour at runtime. Nothing in the file records the difference and nothing in the install output announces it, so the first report of it is a broken service and a teammate who did nothing wrong.

What a lockfile actually pins#

What the teammate needed was not a longer requirements file. It was a record of the installation that actually happened on the day the reranker worked — every package that ended up on disk, rather than only the ones a person thought to name.

To see what such a record would have to contain, look at what installation already computes. A dependency manager is a tool that takes a project's declared dependencies and resolves them into one consistent, installable set of versions. pip install -r requirements.txt is already doing a small, ad hoc version of that job. It reads the five names, walks their metadata, discovers the packages underneath them, chooses a version for each, checks that the choices do not contradict each other, and installs the result. The resolution genuinely happens, and it is genuinely complete. What pip's install step does not do is write any of it down: the moment the last file lands in the environment, the answer it computed is gone, and the only surviving record is the directory of installed files on that one machine.

The fix is to make it write the answer down. A lockfile is a generated file recording the exact resolved version and hash of every dependency, direct and indirect, so the same install can be produced again. It is generated rather than written, which is the reason it can carry things a person never knew to type, and it is committed to the repository alongside the manifest that produced it. A manifest and a lockfile both list packages, which is why they get confused with each other; the manifest states what the project wants, in whatever loose terms are honest — numpy>=1.26, tokenizers>=0.20 — and the lockfile states what one run of the resolver decided those terms mean.

Three things live in that record which a hand-written requirements file has no way of holding. There are dependencies the project never declared and whose names nobody involved has necessarily heard. There are restrictions on which machine a given entry even applies to. And there is a hash of each downloaded file, so an install can check that the bytes it received are the bytes that were resolved. Each of those is a separate mechanism, and each answers a failure the others leave standing.

Dependencies you never wrote down#

To see the first of those problems in action, the reranker's requirements file names anthropic, and anthropic is not one package. Installing it installs a small tree, and none of the names in that tree appear in anything the reranker's authors wrote.

Take the smallest slice of that first. The reranker asks for anthropic; anthropic speaks HTTP and asks for httpx, so httpx is installed even though the reranker never mentions it. A transitive dependency is a package a project depends on only because something it depends on directly depends on it — httpx here, arriving through anthropic, on behalf of code the reranker's authors never read.

Now change one thing about that slice: httpx has dependencies of its own. It asks for httpcore, which handles the connection layer, and httpcore asks for h11, a small library that does nothing but implement the HTTP/1.1 protocol as a state machine. The reranker is now three edges away from a package whose name has never been said out loud in the project, and h11 is as capable of breaking the service as anthropic is. The same descent happens on the other branch: pydantic asks for typing-extensions, a backport shim that carries typing features from newer Python releases into older ones, and a change in that shim reaches the reranker through code nobody involved wrote.

Nothing about writing a longer requirements file repairs this, and the reason is not effort. The graph is not visible from inside the project. A person can pin what they know; to pin h11 they would first have to resolve the graph to learn h11 is in it at all — which is why a hand-pinned requirements file and a lockfile are not the same artefact even when every line of the requirements file carries ==.

Three other explanations tend to get reached for at this point, and each of them describes something pip really does, just not something that closes this gap.

First, an == line is a hard constraint rather than a preference. pip installs precisely the version that line names; if two packages in the graph require versions that cannot both be satisfied, the install fails and says so, and no silent upgrade past a pin occurs. The teammate's drift came entirely from the names that were never written down.

The gap is not that a requirements file only works inside one particular virtual environment either. A virtual environment is an isolated directory of installed packages, and the same requirements file installs happily into any environment on any machine. Two environments created minutes apart on one laptop, both installed from that same file, can still end up holding different code, because the resolution runs separately for each of them.

Nor is the gap about network access. A lockfile records what to fetch; it does not contain the packages themselves, so an install driven by a lockfile still contacts a package index exactly as pip install -r does. Installing without a network means keeping the downloaded wheels somewhere local, which is a separate practice that neither file performs.

Rather, what a fully ==-pinned requirements file is missing is one thing, and it is the resolved transitive graph.

One lockfile, two platforms#

Once the resolved graph exists as an actual list, something about it becomes visible that was never visible in a list of five names: not every package in it is wanted on every machine. tokenizers is a compiled package, and a compiled package is not one file. The release that serves a macOS laptop with Apple silicon is a different build from the release that serves an x86 Linux container, and a resolver deciding what tokenizers 0.20.3 means has to decide which of those files it means.

What lets it decide is a platform marker: a condition attached to a lockfile entry restricting which operating system, architecture, or Python version that entry applies to. Markers are also what the reranker's own manifest uses to say that its accelerated inference runtime is wanted on servers and not on laptops.

dependencies = [
    "anthropic==0.39.0",
    "tokenizers>=0.20",
    "numpy>=1.26",
    "onnxruntime>=1.19; sys_platform == 'linux'",
]

That one marked line is enough to make the resolved graph fork. On a macOS laptop the runtime is not part of the graph at all, and numpy is free to take the newest release the manifest allows. On Linux the runtime is in the graph, it has not been rebuilt against numpy 2 yet, and its metadata therefore caps numpy below that release. A resolver working out one universal answer cannot pick a single numpy here, so it records an answer per condition instead — two numpy entries in the lock, both from the same registry, each carrying the condition under which it applies:

An engineer who locks on a laptop, commits, and watches CI install numpy 1.26.4 has not found a bug. Both installs are correct, and they are correct against the same lockfile, because the lock was never one universal answer — it was an answer conditioned on where it lands. The rule worth carrying out of this is a rule about how the lock was made: a lockfile is only reproducible across machines if it was resolved as a multi-platform lock in the first place, or if a separate lock is generated for each platform the project deploys to. A lock resolved for a laptop alone has nothing to say about a Linux container, and the install there will resolve the missing parts fresh.

Three things about that CI install are worth stating precisely, because each of them is a real mechanism that happens to be a mechanism for something else.

The lockfile is not storing a range that resolves differently each time it is read. Ranges live in the manifest, exact versions live in the lock, and those are two files doing two jobs: the manifest says numpy>=1.26 and expects a resolver to decide, while the lock says 2.1.3 under one condition and 1.26.4 under another and expects an installer to obey.

The index is not serving different available versions to different operating systems. macOS and Linux talk to the same package index and see the same project with the same list of releases; what differs is which platform-tagged file inside a release the index hands back, which is a fact about the files in a release rather than about the release list.

And a cold cache on CI is not what selected 1.26.4. A package cache decides how quickly bytes arrive and whether the index has to be contacted at all. It has no vote in which version the lock resolves to, so a CI runner that has never cached anything installs exactly the entry a warm one would.

Pinning the bytes, not just the number#

The lock, not the cache, is what determines versions — and that lock can fork by platform marker, so two machines can legitimately land on different resolved versions. Given that, a second question opens underneath the first: on any one machine, is the file that arrived the file the lock is talking about? A version number is a label a publisher controls, and the number alone says nothing about the bytes underneath it.

Dependency hash pinning is the recording of a cryptographic hash of each package's exact file alongside its version, so an install can be verified to match, rather than merely resemble, what was resolved. The lockfile stores a hash of the artifact that the resolution selected; the installer recomputes that hash over the bytes it downloads, compares, and refuses to continue if the comparison fails.

[[package]]
name = "anthropic"
version = "0.39.0"
source = { registry = "https://pypi.org/simple" }
wheels = [
    { url = "https://files.pythonhosted.org/packages/../anthropic-0.39.0-py3-none-any.whl", hash = "sha256:8b2a...c41f", size = 198412 },
]

Now take the case the definition only described. An install on a fresh machine asks the index for anthropic 0.39.0, and what comes back is not the file the lock recorded — a release re-uploaded under the same number, or a mirror in front of the index serving a substituted artifact. The version number matches. The digest does not.

error: Failed to prepare distributions
  Caused by: Failed to fetch wheel: anthropic==0.39.0
  Caused by: Hash mismatch for `anthropic==0.39.0`

Expected:
  sha256:8b2a...c41f

Computed:
  sha256:57d9...0e6b

The install stops there. It names the package and version it was working on, the digest the lockfile recorded, and the digest it computed from the bytes it actually received, and it does not proceed to the next package. What matters about where that check sits is that it fires on download, before anything is unpacked into the environment and long before any of the reranker's own code imports it — the substituted file never becomes code that runs.

That is the whole of what a hash buys, and the boundary is worth naming while the mechanism is fresh. A hash proves that the bytes on this machine are the bytes the lock recorded. It has nothing to say about a resolver that legitimately chose a different version for a different platform, and nothing to say about a graph that was never resolved for the platform in question, so the failures of the sections before this one survive hash pinning intact and unaffected. A hash establishes that you got the file the lock names; it cannot establish that the lock names the file you wanted.

Telling the checker what a value should be#

Inside the reranker, past all of that, is the function everything else exists to run.

def score(passage, temperature):
    raw = call_model(passage, temperature)
    return raw["relevance"]

While that function has one caller, sitting twenty lines above it in the same file, its signature costs nothing to read. Once it is called from three places — the request handler, a batch job, an evaluation script — every caller has to open the function body to learn what temperature is allowed to be and what comes back, and every caller does that reading again after the next change to the body.

What the signature is missing is a way to say those things in the signature itself. A type annotation is a hint attached to a variable, parameter, or return value stating the type it is expected to hold. A static type checker is a separate program that reads those hints across a codebase and reports usage inconsistent with them, without running any of the code it reads. Annotating and checking are two jobs and two tools; the interpreter records annotations and otherwise ignores them, and a checker like mypy is what turns them into something that can fail.

def score(passage: str, temperature: float) -> float:
    raw = call_model(passage, temperature)
    return raw["relevance"]

Watch what that buys when one of those three callers gets it wrong. The reranker's evaluation script reads its settings out of a small config file and calls the function with what it found there, where temperature arrived as text and was never converted.

score(passage, temperature=settings["temperature"])  # settings["temperature"] is "0.7"

eval.py:14: error: Argument "temperature" to "score" has incompatible type "str"; expected "float"  [arg-type]
Found 1 error in 1 file (checked 1 source file)

Change one thing about that call — the script converts the setting to a float before passing it — and the same checker, over the same annotation, reports nothing at all.

Success: no issues found in 1 source file

One annotation caught one call and cleared another, without the program running either time. That is the entire trade being made when a signature gets annotated: a claim written down once, in the place a reader looks first, that a tool can hold every call site against.

When the checker has stopped checking#

But the reranker's real scoring function does not take a float. It takes the whole request payload — the query, the list of passages, a temperature, a top-k cut-off, a couple of flags — and while that function is being written, the fastest honest-looking way to say so is to say it is a dictionary of strings to whatever.

def score(payload: dict[str, Any]) -> float:
    return payload["temperature"] * 2

score({"query": q, "passages": ps, "temperature": "0.7", "top_k": 3})

mypy reports nothing here. Not a warning, not a note — the file is clean, and it stays clean through every future edit to that function body.

The Any type is a special annotation telling a checker to skip checking a value entirely, so operations on it are never flagged even when they are wrong. It is not a placeholder meaning unknown, so be careful; it is an instruction meaning stop. Indexing a dict[str, Any] yields an Any, because that is what the annotation says the values are. Multiplying an Any by 2 yields an Any, because a checker asked to skip a value has nothing left to compare against int.__mul__. Every expression that touches the payload inherits the same silence, and the silence spreads outward through the function rather than staying at the annotation.

The -> float is the place most readers expect the silence to break, and following the rule through is what shows why it does not. Any is assignable in both directions: any value at all can be passed where an Any is expected, and an Any can be used anywhere a specific declared type is expected. Returning an Any-typed expression from a function annotated -> float therefore satisfies the checker exactly as a real float would, so the annotation is still present, still read on every run, and still incapable of flagging anything, because the value arriving at it was declared beyond checking several lines earlier.

What happens instead happens at runtime, and it is worse than a crash. '0.7' * 2 is valid Python: multiplying a string by an integer repeats the string. The function returns '0.70.7', no exception is raised, and a string that looks vaguely like a number travels on into whatever sorts the reranker's results.

The structure that let the string through has a name. An unconstrained mapping is a dictionary annotated with a permissive value type such as Any, so a checker cannot verify which keys exist or what shape the values behind them have. Key presence is a separate limit, and it is worth keeping the limits apart: a plain dict[str, str] would not have told a checker which keys exist either, because standard checkers do not track the keys of a plain dictionary at all. What Any removed is the checking of the values, and that removal is specific to the annotation chosen here.

The cost of that is not the bug itself but where the bug surfaces. An annotation that looks like it named a type switched checking off, and the resulting failure appears at whatever call site first passes the wrong shape, in whatever module that call site lives in, long after the signature that permitted it was written and reviewed.

Three ways to give a payload shape#

Giving that payload an actual shape means choosing among three constructs Python offers for the job, and the three are worth naming together before any one of them is opened: a TypedDict, a dataclass, and pydantic's BaseModel.

{
    "query": "how do i rotate an api key",
    "passages": ["Key rotation lives under Settings...", "Billing runs monthly..."],
    "temperature": 0.7,
    "top_k": 3,
}

Each of the three is demonstrated below on the reranker value that actually sits where that construct belongs: the incoming payload dictionary for TypedDict, a ScoreResult the reranker's own code builds for dataclass, and a ScoreResponse parsed out of the model's reply for BaseModel. Those are deliberately three different values. One value carried through all three would have to pretend to occupy positions it cannot occupy at once — arriving from outside and being constructed internally and being read without reshaping — and the pretence would hide the thing that actually decides between them. What stays continuous across the sections is the reranker and the axis being tested, not a single object.

What each one does to a payload, side by side:

Read the last column downward and you have the actual decision, because the choice between them is not a matter of taste about which reads nicest. A trust boundary is the point in a system where data arriving from outside crosses into code that treats it as validated and safe to use. The question that separates the three is where the value in hand sits relative to that boundary: already inside it and merely read, built inside it by the reranker itself, or crossing it right now.

Describing a dict without changing it#

Start with the value that never crosses the boundary. The reranker's payload sometimes arrives from another function in the same codebase, one that assembled it from arguments it already validated, and the scoring function reads two fields out of it and passes the whole thing on to a client library that wants a dictionary. Nothing about it is reshaped, and nothing about it is in doubt.

class RerankPayload(TypedDict):
    query: str
    passages: list[str]
    temperature: float
    top_k: int

Give that payload the signature payload: RerankPayload and a checker has something to hold call sites against. A typo in a key is now a fact the checker knows about, and so is a value of the wrong type.

rerank.py:31: error: TypedDict "RerankPayload" has no key "temprature"  [typeddict-item]
rerank.py:47: error: Incompatible types (expression has type "str", TypedDict item "temperature" has type "float")  [typeddict-item]

Not one byte of runtime behaviour changed to get that. The object at run time is the same plain dictionary it was before, built the same way, passed to the same library call, costing the same nothing.

That freedom from runtime cost is also where TypedDict stops being useful. If that dictionary was assembled from a request body rather than from trusted arguments, the class describes the shape the dictionary is supposed to have and — checking nothing while the program runs — stays perfectly silent when the shape that actually arrives is something else.

Constructing values you already trust#

Move one step along the axis, to a value the reranker builds rather than receives. The scoring function computes a relevance number for a passage, works out where that passage lands in the ordering, and needs to hand something back that carries all of it. There is no incoming dictionary to describe here, because the value does not exist until this code creates it.

@dataclass
class ScoreResult:
    passage_id: str
    score: float
    rank: int

results = [
    ScoreResult(passage_id=p.id, score=s, rank=i)
    for i, (p, s) in enumerate(ranked, start=1)
]

The decorator generates the constructor, the __repr__ that makes these readable in a log line, and the __eq__ that lets a test compare an expected result against an actual one. Validation would be wasted work at this position, and it is worth being concrete about why: every argument in that constructor call came from the reranker's own code a few lines earlier. p.id came out of the retrieval step, s is the number the scoring function just computed, i is the enumerate counter. There is no outside here, and therefore nothing arriving that could be checked against anything.

The limit is the same fact read from the other side. A ScoreResult will accept a negative score, a rank that is a string, None in a field annotated str, and it will do so silently. If a later change starts passing bad values into it, the annotations on those fields will not be what notices.

Parsing values you don't#

The scoring function does not only produce a ScoreResult. Yet the real problem lies in what it must consume to produce one: something that crossed the trust boundary named earlier — the model's own answer to the scoring prompt, arriving as JSON text that the reranker has never seen before, produced by a system that is under no obligation to be well-formed and occasionally is not.

That is the position the third construct was described for, and the reranker fills it with a pydantic BaseModel.

class ScoreResponse(BaseModel):
    passage_id: str
    relevance: float

raw = completion.content[0].text          # '{"passage_id": "p3", "relevance": 0.9}'
parsed = ScoreResponse.model_validate_json(raw)

parsed.relevance is a float because the class said float and the parse enforced it, not because the model happened to send one. The Pydantic documentation is the authoritative reference for what that enforcement covers and how it is configured, and algomaster.io's Type Hints and Pydantic explainer sets out why type hints plus pydantic end up being the contract layer for exactly this kind of model input and output.

Enforcing something at run time is also what creates two failure modes that neither a TypedDict nor a dataclass could ever have produced, because neither of them looks at a value while the program runs. The incoming data might not match the declared shape at all. And even when it does match in spirit, its values might need converting into the types the fields declare — a distinction that decides what the reranker sees on a large fraction of real model output.

Coercion, not just validation#

In practice, models return JSON that is nearly right far more often than they return JSON that is wrong. A common near-miss is a number that arrives quoted.

{"passage_id": "p3", "relevance": "0.9"}

Parsing that reply through the same ScoreResponse class, with no configuration changed, does not raise.

ScoreResponse(passage_id='p3', relevance=0.9)

The relevance field holds the float 0.9, and the quoted string is gone rather than stored. That conversion — where the quoted "0.9" becomes a float — is type coercion: the conversion of an input value into a field's declared type during validation, such as turning the string '3' into the integer 3. Following the rule that produces it: pydantic v2's default validation mode is lax, and lax mode converts a clean numeric string into the declared numeric type as part of validating it. Rejecting that same input requires opting the field, the model or the individual call into strict mode, which the Pydantic documentation covers as the deliberate alternative to the default. Dave Ebbelaar's Pydantic Crash Course walks through the same behaviour by hand, including how it plays out through nested models where a coerced field sits inside another model.

One other setting tends to get folded into this one while the defaults are on screen, and keeping them apart matters. extra governs what happens to keys in the input that the model never declared: dropped silently by default, kept as attributes under allow, rejected under forbid. It says nothing about how a declared field's value is converted, so a ScoreResponse configured with extra='forbid' still turns "0.9" into a float without complaint, and would only object to an undeclared key sitting beside it. Strict versus lax mode decides whether a declared field coerces; extra decides whether undeclared keys are tolerated; they are independent knobs on independent behaviours.

Lax coercion also has a place where it stops. A relevance of "0.9" converts cleanly because there is one obvious float it means. A relevance of "high" does not, and no amount of leniency invents a number out of it.

What a validation failure carries, and what not to do with it#

{"passage_id": "p3", "relevance": "high"}

pydantic_core._pydantic_core.ValidationError: 1 validation error for ScoreResponse
relevance
  Input should be a valid number, unable to parse string as a number [type=float_parsing, input_value='high', input_type=str]

What that reply raised is a ValidationError, the exception a validation library raises when input data does not match a model's declared shape, carrying the location, message, and offending value for each failed field. Structured rather than prose, an entry per failure, reachable as data.

[
    {
        "type": "float_parsing",
        "loc": ("relevance",),
        "msg": "Input should be a valid number, unable to parse string as a number",
        "input": "high",
    }
]

Field by field, that says which field failed, what kind of mismatch it was, and what arrived. A parse failure rate can be split by type because float_parsing and missing are different codes, and the fix for a model that returns prose instead of JSON has nothing in common with the fix for a model that gets one field's type wrong.

Then there is the line almost every handler writes on first contact with this exception.

except ValidationError as e:
    logger.error(str(e))

Stringifying the exception renders every entry, including the input for each failing field, which is how input_value='high' came to be in the raised exception above. In that reply the offending value is the word high and nothing is lost by writing it down. In the reranker's other input path it is a request body from a webhook, and in this path it is whatever the model echoed back, which routinely includes a fragment of the user's own text, an identifier, a token the prompt was carrying, or a chunk of a retrieved document. Whatever it is, logger.error(str(e)) copies it verbatim into the log store, where it lands under a different retention policy and a different access list than the request it came from.

except ValidationError as e:
    logger.error(
        "rerank parse failed",
        extra={"failures": [{"loc": err["loc"], "type": err["type"]} for err in e.errors()]},
    )

That keeps everything an on-call engineer or a weekly breakdown actually reads — which field, which failure code — and leaves the value itself out of the record, or in a redacted summary if the shape of the value matters.

Two things about that handler are worth pinning down, because both get read into it. Logging is a side effect and nothing more: logger.error(str(e)) does not swallow the exception, does not re-raise it and does not alter what the except block does next, so the fault in the original line is disclosure into the log store rather than any change to the handler's control flow. And there is nothing special about the catch itself. In pydantic v2 ValidationError derives from ValueError and therefore from Exception, so a broader except ValueError would have caught it identically; naming it in the except clause buys specificity about what is being handled, not the ability to handle it.

The other half of the fix a handler tends to grow at this point is a retry: catch the ValidationError, call the parse again, and hope. Following that through settles it. The exception came from parsing bytes the reranker already holds in memory — raw is a string variable, it has not changed, and nothing about the parser is time-dependent — so a second parse of those same bytes fails identically, with the same code, on the same field. A retry only ever helps where the second attempt has different input, which here means issuing a fresh model call with a corrected or reinforced prompt rather than re-running the parser over the answer already received.

Written the blind way, that retry has a price. It spends the latency and the tokens of a second completion that could not have succeeded, on every failure, including the deterministic ones like a missing required field or an enum value the model will produce again. Where the handler retries the whole request rather than only the parse, it also repeats whatever the request already did — a second retrieval, a second write, a second charge against a rate limit — for an input that was never malformed.

Choosing between the three#

Now that the cost of catching shape errors inside handlers is clear, the question is how to avoid paying it: pick the right construct at the point the value first enters the code. The choice turns on one axis, where the value comes from, and on that axis it decides cleanly.

A runtime failure carrying field-level diagnostics is worth paying for at the bottom row of that table and is wasted money above it, which is the whole of the rule. Validating a value the reranker itself constructed buys a check on code that already had the answer; describing a value that arrived from a model buys silence exactly where silence is dangerous.

There is also the case the rule does not force at all. A dict[str, Any] is a perfectly honest signature for a function that is a genuine pass-through — one that accepts a payload, forwards it to something else, and never indexes into it. The earlier failure was never about Any as such. It was about indexing into a value whose shape the checker had been instructed not to look at, and a function that never indexes never reaches that instruction.

Cells that remember more than the file shows#

Every one of those constructs constrains the code as it is written, and a checker reads the file and nothing else. That is enough as long as the file is the whole record of what ran. A notebook — an interactive document interleaving code cells, their output and prose, where cells can be run individually and in any order rather than only in the order they sit in the file — is where that assumption comes apart.

The reranker's accuracy number — the one on a slide in last quarter's deck — came out of a notebook a researcher ran some weeks ago.

That last property, the running out of order, is the one that matters here. Because cells run on demand, what a notebook currently displays is a record of which cells were actually executed, in whatever order somebody happened to execute them, and the order they appear in the file has no authority over any of it. Kernel state is the set of in-memory variables and definitions a notebook's execution process holds at a given moment, shaped by whichever cells have actually run rather than by their position in the file.

Notebook, the day the number was produced

# In [7]
lowered = [p.lower() for p in passages]

# In [12]
accuracy = evaluate(reranker, lowered, labels)
print(accuracy)
# Out: 0.68

Cell 7 ran, cell 12 ran, 0.68 appeared under cell 12, and the number went into the deck. So far the file and the kernel agree with each other.

Now change one thing, the way the following week actually goes. The researcher decides the lowercasing belongs inside the retrieval helper instead, deletes cell 7, and carries on working in the cells further down without ever re-running cell 12.

# In [12]
accuracy = evaluate(reranker, lowered, labels)
# Out: 0.68

The cell that bound lowered is not in the file any more. The output that depended on it is still on the screen, still says 0.68, and is still what anybody opening the notebook reads first. The kernel held that variable in memory the whole time, so nothing broke and nothing warned. A saved notebook records what some kernel session produced; the file records what the cells currently say; and nothing forces those two records to describe the same computation.

The test that exposes it#

The only way to find out whether a notebook's displayed output reflects the file in front of you, rather than a kernel remembering something the file no longer says, is to take the memory away. Restart & Run All is a notebook command that discards the current kernel state and re-executes every cell in top-to-bottom file order, exposing any result that depended on hidden or out-of-order state.

Running it on the eval notebook gets as far as cell 14.

NameError                                 Traceback (most recent call last)
Cell In[14], line 1
----> 1 accuracy = evaluate(reranker, lowered, labels)

NameError: name 'lowered' is not defined

Read that for exactly what it says. A NameError means the interpreter reached a name that has no binding in the current session — a fact about the file as it stands today, executed from the top with nothing in memory. Two other explanations are commonly reached for and neither produces this error. Python does not silently unbind names under memory pressure; a process running out of memory raises MemoryError or is killed by the operating system, and in both of those cases the names it held remain bound right up until it stops existing. A dependency that has moved since the notebook last ran also fails differently: a renamed or removed library attribute raises AttributeError, a changed signature raises TypeError, and a changed algorithm quietly returns a different number — none of them announces a name that was never bound.

What the failure establishes is narrow and worth stating exactly. It does not say the reranker's accuracy is wrong, and it does not say 0.68 was ever a lie. It says the number in last quarter's deck has no reproducible provenance: there is currently no run of this file that produces it, and until there is, the number rests on the memory of a kernel session that ended weeks ago. Restart & Run All is the cheapest available test of whether the artefact in the repository still produces the artefact that was reported, which is why it is the first thing to do to a notebook somebody is about to trust.

When to leave the notebook behind#

The eval number is now wanted monthly rather than once, which changes what the file has to be. The execution order that belongs to a person's session has to become a fact about the file. A repeatable script is a plain Python file with a single fixed top-to-bottom execution order, so running it twice from a clean environment performs the same steps in the same sequence both times.

Getting there has an order, and the order is what protects the number.

Run Restart & Run All first, and compare its output against the value already in the deck rather than merely checking that the notebook ran. A clean run that produces 0.71 is a different situation from a clean run that produces 0.68, and only one of those leaves the deck standing.

If the number doesn't match, expect the cause to be one of a small set, and look for it before touching the notebook's structure: a cell edited after it was last run, a variable defined in a cell that has since been deleted, an absolute path pointing at a directory on the researcher's own laptop, or a package that has moved underneath the notebook since the day it ran. Each of those is a small, local repair, and each is far cheaper to make in the notebook than in a rewrite.

Then, before any rewrite, pin the environment, fix the input snapshot, and fix the random seed. The reason to do that first is a reason about evidence: a rewrite that also changes the reported number leaves nobody able to say whether the rewrite is wrong or the original computation was, and there is no way back to the answer from there.

The smallest change that makes the number trustworthy is that list plus one more thing, and it does not require a rewrite at all: a pinned environment, a fixed input snapshot, a deterministic seed, and a final cell that asserts the computed value against the number already reported. A notebook that ends in a failing assertion is telling you something the moment it fails; a notebook that ends in a printed float is telling you something only if somebody happens to read it. Once that assertion holds, turning the notebook into a repeatable script becomes a sensible next step — a separate one, taken against a number that has already been shown to reproduce.

Reading a traceback that starts inside someone else's code#

The reproducible notebook keeps the environment from being the surprise; it does nothing about the data the reranker has never seen. The reranker is running in production, the model returns something malformed, and the parse raises inside a library nobody at the company wrote. What lands in the log is a traceback: the report Python prints when an exception propagates uncaught, listing the sequence of stack frames that were active at the point it was raised. A stack frame is the record of a single function call inside that report, showing the file, the line and the function where execution stood at that point in the call chain.

Traceback (most recent call last):
  File "/srv/reranker/api.py", line 88, in post_rerank
    return handle(request.json())
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/reranker/handler.py", line 42, in handle
    parsed = ScoreResponse.model_validate_json(raw)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/reranker/.venv/lib/python3.11/site-packages/pydantic/main.py", line 656, in model_validate_json
    return cls.__pydantic_validator__.validate_json(json_data, strict=strict, context=context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for ScoreResponse
relevance
  Input should be a valid number, unable to parse string as a number [type=float_parsing, input_value='high', input_type=str]

Read top to bottom, that is a call chain in the order the calls happened: post_rerank called handle, handle called model_validate_json, and inside that call the exception was raised. The bottom line names the exception and the frame directly above it is where it came out, which is why the eye goes there first and why the eye is usually in the wrong place. That frame sits in site-packages, in a file whose path contains the virtual environment directory, and nothing in it is editable by the person reading it.

The frame worth finding is the last one whose path belongs to the reranker — handler.py, line 42 — because that is the call boundary: the final line of code the team owns before execution crossed into somebody else's. It names the value that was passed in, raw, and it is the line a smaller reproduction gets built around. The frames below it describe how the library reacted to that value, which is useful for understanding the failure and useless for changing it.

Two ways one exception leads to another#

The reranker's handler does not let that exception escape. It catches the ValidationError and raises the reranker's own error type, so that everything upstream deals with one failure vocabulary rather than with whichever library happened to be in the call chain. What is worth seeing is that the same catch-and-reraise, with the same intent, prints two different tracebacks depending on a single word.

Shared between the versions below: the same first exception, raised by the same parse of the same malformed reply; the same except ValidationError as e: block; the same intention to raise a RerankFailure in its place.

try:
    parsed = ScoreResponse.model_validate_json(raw)
except ValidationError as e:
    raise RerankFailure("bad model output")

Traceback (most recent call last):
  File "/srv/reranker/handler.py", line 42, in handle
    parsed = ScoreResponse.model_validate_json(raw)
pydantic_core._pydantic_core.ValidationError: 1 validation error for ScoreResponse
relevance
  Input should be a valid number, unable to parse string as a number [type=float_parsing, input_value='high', input_type=str]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/reranker/api.py", line 88, in post_rerank
    return handle(request.json())
  File "/srv/reranker/handler.py", line 44, in handle
    raise RerankFailure("bad model output")
reranker.errors.RerankFailure: bad model output

try:
    parsed = ScoreResponse.model_validate_json(raw)
except ValidationError as e:
    raise RerankFailure("bad model output") from e

Traceback (most recent call last):
  File "/srv/reranker/handler.py", line 42, in handle
    parsed = ScoreResponse.model_validate_json(raw)
pydantic_core._pydantic_core.ValidationError: 1 validation error for ScoreResponse
relevance
  Input should be a valid number, unable to parse string as a number [type=float_parsing, input_value='high', input_type=str]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/srv/reranker/api.py", line 88, in post_rerank
    return handle(request.json())
  File "/srv/reranker/handler.py", line 44, in handle
    raise RerankFailure("bad model output") from e
reranker.errors.RerankFailure: bad model output

The two versions differ in one word, and that word is the difference between the two kinds of chaining. Implicit exception chaining is Python's automatic linking of a new exception to whichever exception was already being handled when the new one was raised, printed as During handling of the above exception, another exception occurred. Explicit exception chaining is the deliberate linking of a new exception to a specific cause with raise ... from ..., printed as The above exception was the direct cause of the following exception.

The difference between those seam lines is a difference in what the code claimed, and it changes how the frames above the seam should be read. Implicit chaining reports a coincidence of timing: some exception was being handled, and while it was, another one was raised. Very often that second exception comes out of cleanup — a rollback that fails, a connection close that raises, a formatter that trips over the very value that caused the original problem — and in those cases the block above the seam is the aftermath rather than the diagnosis. Explicit chaining reports a judgement somebody made: this earlier exception is why this later one exists. Reading which phrasing sits at the bottom of a log is therefore a fast way to decide whether the frames above it are the fault or the debris around it.

Three things that phrasing does not indicate are each worth knowing as facts about how Python behaves.

contextlib.suppress does not produce either seam. It swallows the exception it is given entirely; nothing is re-raised, so there is no second exception for a first one to be attached to, and a suppressed failure leaves no traceback at all.

Neither seam indicates lost frames. A chained traceback prints every frame of both exceptions, which is exactly what the blocks above show — the parse frames appear in full, above the seam, alongside the handler frames below it. Crossing into a compiled extension does not remove Python frames from the chain.

And neither phrasing has anything to do with exceptions raised in parallel. Failures from sibling tasks or threads that were never handling one another surface as an ExceptionGroup, printed as a set of separate tracebacks under a group header rather than joined by a seam line. Seeing either seam at all establishes that both exceptions met in one stack, one raised while the other was being handled or because of it.

A checklist for a new Python AI project#

Chaining phrasing is the last of the pieces; the rest — lockfiles, annotations, parsing at the boundary, notebooks that produce a number twice — arrived earlier in the chapter. Gathered together they make a pass over a project. Each line below is answerable by looking at the project in front of you.

This is what the reranker needed, in the vocabulary the sections above built. A lockfile against a hand-pinned manifest, so a second machine installs the graph rather than re-resolving it. A real shape against Any, so a checker has something to hold call sites to. Three ways of writing that shape down, chosen by where the value sits relative to the trust boundary. A script's fixed order against a notebook's kernel state, so a number can be produced twice. And a traceback's chaining phrasing read for what it claims, so an on-call engineer looks at the cause instead of the debris.

All of it is overhead, and it pays for itself in exactly one situation: something happening a second time. A second install on a second machine, a second call carrying data the first call never had, a second month of the same evaluation, a second engineer reading the same failure at three in the morning. Against a script written to answer one question this afternoon and deleted tomorrow, it buys nothing at all, and the honest reading of a throwaway file is that none of this is worth doing to it.

Two of the decisions here do not resolve into a rule, and the useful thing is to say so rather than to pick a side. Hash verification costs a check on every download and buys protection against a substituted or re-uploaded artifact, and whether that trade is worth making depends on what the project installs and where it installs from. How strict a pydantic model should be depends on which caller it is standing in front of: lax coercion turns a quoted number into a usable one, and it also turns a caller's sloppiness into your problem quietly. Both of those are measurements against a specific system rather than defaults, and the reranker's answer is not automatically anybody else's.