Published and effective knowledge cutoffs

A model is trained once, on a large body of text collected up to some point in time, and when training ends its weights are frozen; nothing that happens in the world afterwards changes them, and no later event writes itself into them. Everything the world settled after that point — the current name of an API, this quarter's pricing, the policy edit someone published on Tuesday — is absent from the weights and has to reach the model through the prompt instead, which is the arrangement the rest of this topic works out. The published cutoff exists because there is otherwise nothing to reason from about the age of what a model tells you: the weights carry no sources and no timestamps, so an answer about a library release reads the same whether the model absorbed ten thousand pages on the subject or none.

Two things follow from that freezing, and only the first is obvious. Training is the process that writes into the weights: it adjusts a very large set of shared parameters, repeatedly, so that the text it was shown becomes a more likely continuation of itself. When the run ends, there is no such process running any more. Nothing later reaches the parameters — not a news article, not a release note, not the question you are about to ask. The second consequence is the one this topic turns on. Because the writing was done by adjusting shared parameters rather than by filing documents, the run leaves behind no inventory of what it saw: no field holding a date, no count of documents, no list of subjects covered. A cutoff is therefore a fact about which text was available to that run, held entirely outside the model, and there is nothing inside the model that corresponds to it.

This chapter is about the gap between that published date and what a model actually knows. The date turns up as a single line on a provider's model page, reading something close to training data up to approximately November 2024, and it sits in the documentation next to the context window and the output limit — numbers that are exact, contractual and enforced by the serving stack, whose air of precision it inherits by adjacency. Exceed a context window and you get an error. The cutoff is an estimate about a collection of text, nothing enforces it in either direction, and the chapter's work is to say what it does support, how to record it so that it cannot drift, and how to measure the effective cutoff underneath it for the subjects you serve.

The date describes a collection process, not a state of knowledge#

A published cutoff is a statement about a data pipeline. It says, approximately, when the crawls and dumps that fed pretraining — the long first training run over general text, which is where nearly all of a model's world knowledge comes from — stopped being collected. Between that fact and the thing engineers usually infer from it, that the model knows what happened up to this date, sit deduplication, quality filtering, sampling weights, and the sheer accumulation dynamics of written text.

That last one does most of the damage. Text about any event or release keeps being written for years afterwards: the launch post, then the tutorials, then the Stack Overflow answers, then the blog posts comparing it to its successor, then the migration guides. A library released three months before the cutoff has three months of accumulated coverage in the training data. The same library, seen by a crawl run three years later, has three years. Near the cutoff, every topic is thin, because the world has not finished writing about it yet. The effective boundary of a model's knowledge is therefore not a line, it is a slope, and the slope starts well before the published date.

This is measurable rather than theoretical. The COLM 2024 paper Dated Data: Tracing Knowledge Cutoffs in Large Language Models estimated per-resource effective cutoffs by measuring which version of a document a model assigns higher likelihood to, and found that effective cutoffs frequently precede reported cutoffs — in a number of cases by a year or more — and that they differ substantially between subdomains of the same model. Two mechanisms explain most of it. Deduplication against earlier snapshots systematically removes the newest copy of a document and keeps an older one, and a crawl performed in any given month is dominated by documents written long before that month, so the tail of recent material is small before any filtering happens at all.

Those two mechanisms are worth putting in order, because as a chain the slope becomes something a reader can rebuild rather than take on trust. Text about an event keeps being written for years, so the quantity of text about any event grows with the time since it happened. A crawl performed at a given moment therefore samples a world in which last month's releases have almost nothing written about them and five-year-old ones have a great deal, which is the second of the two mechanisms above stated as a consequence rather than as an observation. Deduplication then thins the recent slice a second time, because it compares each document against earlier snapshots and the copy it discards is the newest. Neither step involves anybody deciding to under-represent recent material. The thinness is produced by how text accumulates, and then again by a filter built for an unrelated purpose.

Two predictions fall out of that chain, and both can be checked against material already in this chapter. If thinness near the cutoff is a matter of how much has been written, it should be worst where the total quantity written is smallest: a subject whose entire public footprint is one changelog, one README and a few dozen forum posts is thin at every date and thinnest of all near the edge, while a subject the whole trade press covers passes whatever threshold matters within weeks of happening. And if that holds, a revision trained on a snapshot collected months later should improve heavily-written subjects far more than narrow ones, because the extra months added a great deal of text about the former and almost none about the latter. The scenario at the end of this chapter is that second prediction landing, on a team whose product depended on the domain that did not move.

The per-revision variation is a second, independent source of surprise. Revisions inside a family are often trained on different data snapshots, with different filtering and different mixture weights, and the relationship between two of them is not a superset relationship. A revision can be newer overall and worse on a specific subject, because a filtering change dropped a source that the earlier snapshot happened to include heavily. Teams discover this when an upgrade that improved every headline benchmark quietly regresses one narrow capability that the product depends on, and the release notes contain nothing that would have predicted it. Treating each dated revision as its own artefact — its own recorded cutoff, its own probe results, its own evaluation run — is what makes that discoverable instead of mysterious.

The step that makes a superset relationship feel natural is worth naming, because it holds for almost every other artefact an engineer versions. A newer index contains the older index plus whatever arrived since; a newer release contains the previous release's code plus a diff. Neither pattern is guaranteed here. A training run's input is a body of text, and unless a provider states that a revision continued from its predecessor's weights, there is no ground for assuming that anything the earlier revision learned was carried into the later one. Where nothing carries forward, two revisions of one family are two independent readings of two overlapping piles of text, and the later reading can be poorer on any subject the second pile happened to under-represent. "Newer" orders the collection dates. It does not order what is known.

Each row of the table below pairs an assumption with what is actually the case, and the left column reads better as a set of reasonable inferences than as a list of mistakes. Every one of them would be sound for a number that something enforced. A context window is enforced: exceed it and the request is rejected, so the boundary announces itself and nobody has ever had to measure where it sits. The habit that builds is that a documented limit is self-reporting, and the cutoff is documented in the same list as the limits that are. Nothing enforces it — not the serving stack, which never consults it, and not the model, which has no representation of it — so being wrong about a cutoff produces no error, no rejection and no log line. That is why the cost of each wrong assumption lands in the third column, quietly, on whoever reads the answer.

Recording something you do not fully trust#

Writing down a number you have just been told not to trust sounds like a contradiction, and it is worth resolving before doing the work. The date does not support the inference people want from it, which is that everything before it is known. It supports a narrower one. Something that was never in the crawl cannot be in the weights, and the date is the provider's estimate of when the crawl stopped, so it bounds what could possibly be in there. That makes it a ceiling with soft edges rather than a floor — worth recording, and worth recording carefully, because it moves between revisions without appearing in any diff of yours.

The narrow, mechanical task is this: go to the provider's model documentation, find the approximate training-data cutoff for the exact model revision you call, and record it next to that revision in your own repository. Not the family, not the alias — the revision. Aliases that resolve to "latest" are convenient in a console and hostile in a system where you are trying to reason about what the weights contain, because the answer changes underneath you without a deploy.

A registry file is enough, and it works because it is checked in and reviewable:

# models.yaml — one entry per model revision we are permitted to call.
# `training_cutoff` is copied from the provider's docs. It is an assumption,
# not a guarantee; see effective_cutoff_probe.py for what we measured.
revisions:
  - id: atlas-4-1-20250312
    role: primary
    training_cutoff: 2024-11
    cutoff_source: https://provider.example/docs/models#atlas-4-1
    cutoff_recorded_on: 2025-04-02
    notes: >
      Provider states "approximately November 2024". Post-training updates
      are not dated anywhere in the docs.

  - id: atlas-4-2-20251007
    role: candidate
    training_cutoff: 2025-06
    cutoff_source: https://provider.example/docs/models#atlas-4-2
    cutoff_recorded_on: 2025-10-14
    notes: >
      Same family as 4-1, seven months newer. Do not assume this is a
      superset of 4-1's knowledge; it is a different snapshot.

Then make the deployment unable to drift away from it:

import yaml

with open("models.yaml") as fh:
    REVISIONS = {r["id"]: r for r in yaml.safe_load(fh)["revisions"]}


def resolve(model_id: str) -> dict:
    """Fail loudly if we are calling a revision nobody recorded a cutoff for."""
    if model_id not in REVISIONS:
        raise RuntimeError(
            f"{model_id} is not in models.yaml. Record its training cutoff "
            f"from the provider docs before shipping it."
        )
    if model_id.endswith(("-latest", "-preview")):
        raise RuntimeError(
            f"{model_id} is a moving alias; pin the dated revision instead."
        )
    return REVISIONS[model_id]

The alias check earns its place the first time an incident report ends with "nothing changed on our side". A -latest pointer resolving to a new revision on the provider's schedule is a production change that never appears in your deploy log, never gets a rollback, and never shows up in a bisect over your own commits. It is also the change most likely to move a cutoff, because a new revision is exactly what a new snapshot arrives with. Pinning the dated revision costs a line in a config file and converts an unexplainable behaviour shift into a change with an author and a date.

Probing the effective cutoff per domain#

The registry gives you the vendor's claim. Measuring the slope underneath it takes a probe, and the probe is worth building once because it is the only way to find out whether a newer revision bought you anything on the topics you actually serve. Pick a dozen facts per domain you care about, each with a known date of first public existence, spread across the two years around the published cutoff. Ask about each one in a form that has a checkable answer, and record which ones come back right.

PROBES = [
    # (domain, question, first_public, expected_substring)
    ("client-lib", "What is the current name of the event subscription "
                   "method on the SDK's Session class?",
     "2024-08", "subscribe"),
    ("client-lib", "Which SDK major version removed stream_events?",
     "2024-08", "3.0"),
    ("cloud-pricing", "Which instance family replaced the m6 series?",
     "2025-01", "m7"),
    ("regulation", "What reporting frequency did the 2025 amendment set?",
     "2025-03", "quarterly"),
]


def probe(model_id: str) -> list[tuple[str, str, bool]]:
    results = []
    for domain, question, first_public, expected in PROBES:
        answer = call_model(model_id, question)  # no retrieval, no tools
        results.append((domain, first_public, expected.lower() in answer.lower()))
    return results

The shape of that output follows from the slope rather than from a taste for granularity. If coverage ended at a line, a probe would only have to find the line, and one number per revision would do. Because it decays at different rates on different subjects, a single figure would average a domain the model handles unaided against one where it is years behind, and the average would describe neither. Spreading the facts across the two years around the published date matters for the same reason: a set drawn entirely from the months just before the date samples only the thinnest part of the band, and makes every revision look uniformly bad.

Two details make the probe worth maintaining rather than running once. The questions must have short, checkable answers — a method name, a version number, a rate — because a probe scored by human judgement will not be re-run, and a probe that is not re-run is a one-off investigation rather than an instrument. And it belongs in the same evaluation pass you already run when qualifying a revision, so that "how stale is this model on the four domains we serve" is answered before a candidate is promoted rather than after a customer notices. The output is not a pass or fail; it is a per-domain hit rate that tells you which subjects need retrieval and which the weights handle unaided, which is the same decision the sixth chapter of this topic formalises.

That rate has a use beyond routing. A domain where the model scores two of twelve is a domain where nobody should be reviewing the assistant's unsourced answers and calling them plausible, because plausibility is exactly what the model produces there. Publishing the per-domain numbers to the people who use the system does more for their calibration than any amount of interface copy about limitations.

What the date is silent about#

It is not a boundary the model enforces, and the next chapter is entirely about that. It is also not the model's sense of the current date. Many deployments inject a date into the system prompt without ever checking what the model does in its absence, and a model with no injected date will still reason confidently about what is "recent" and what is "upcoming", anchored on whatever its training distribution suggests. Two systems calling the same revision, one of which stamps today's date into the prompt and one of which does not, will disagree about whether a 2025 release is in the past.

Both halves of that are consequences of the first section. A boundary the model could enforce would have to exist somewhere it could be consulted, and the training run left no inventory to consult — no date, no document list, no record of which subjects were covered. So a question about something past the edge is not met by a check that fails. It is met by the same machinery every other question is met by, which is why the next chapter opens on a fluent answer rather than on a refusal. The current date is the same absence seen from the other side. Nothing in the computation that produces a response reads a clock, so the only way a date reaches the model is as text somebody put in front of it. A deployment that never injects one has not left the model neutral about the present; it has left it to infer the present from whatever its training distribution suggests, which is a guess it will make confidently and never flag.

It is not a statement about post-training either. Instruction tuning, preference optimisation and safety work happen after pretraining, on separately collected data that providers rarely date. A revision can change how it behaves — how it hedges, when it declines, what format it prefers — with no movement in the published data cutoff at all. If you are tracking behaviour regressions, the cutoff is not the field to look at.

And it has nothing to say about your own data. Your refund policy, your customer records, yesterday's deploy log and the internal RFC someone wrote this morning were never in any crawl, at any date. No amount of model recency reaches them. That is a separate problem with a separate solution, and it is worth keeping mentally distinct from staleness, because "the model is out of date" and "the model has never seen this" feel identical from the outside and have entirely different fixes.

The honest limitation is that the effective cutoff is not fully knowable. Probing gives you a per-domain estimate on the questions you thought to ask, and the domains you did not probe stay unmeasured. A provider cannot publish a per-subdomain map either, because deriving one would mean characterising trillions of tokens of text against every possible topic.

So the published date is worth exactly what it is: the only externally supplied signal about the weights, worth writing down against a pinned revision, and unable to support the inference people want to make from it. The registry gives you a claim with a source and a date you recorded it on; the probe narrows that claim to a per-domain hit rate on the subjects your product serves; and everything downstream should be designed as though the answer might be stale in ways nobody has measured, because on the domains you did not probe it is.

Which leaves a question the registry cannot answer. You now know, roughly, where the weights stop. The model does not — there is no field inside it holding a date, no internal flag that fires when a question runs past the edge of what it saw. What happens when you ask anyway is the subject of the next chapter, and it is not a refusal.