Inclusion criteria, not interest, decide what stays on the list

A lab published a model on a Tuesday at 14:00 UTC. By Thursday evening the saved-links folder held 41 items, all of them collected in the ordinary way: something looked interesting, it went into the folder for later. Reading them in one sitting on Friday morning produced an unpleasant discovery. Six of the 41 were artefacts — the model card, the pricing page, the API changelog entry, a paper the card referenced, one independent benchmark run with the harness published alongside it, and a long write-up from an engineer who had put the model through his own test set and shown the outputs. The remaining 35 were commentary on those six. Nineteen of them quoted the same benchmark table from the model card. Eight quoted each other. Four were summaries of the write-up, and one of those four had got a number wrong by transposing two rows, which then propagated into two of the others.

The time cost of reading 41 items instead of six is about ninety minutes, which is annoying and survivable. The judgement cost is worse and less visible. Reading the same claim nineteen times produces a feeling of corroboration that nothing in the evidence supports, because the nineteen sources share one origin, and repetition is the mechanism by which a self-reported benchmark number becomes something you believe you have seen confirmed. The folder was not a sample of what the field thought. It was one lab's press release with an echo attached, weighted by how many people in one social graph happened to be awake.

What has to be kept up with is smaller than the firehose implies#

That folder is one instance of a general obligation, which is that shipping a product on top of models built by other companies commits you to tracking things you do not control. Staying current, for an engineer in that position, means knowing four things at any given moment: which models exist and what they cost, which APIs and defaults are changing or being retired underneath you, which capabilities have appeared that your product could plausibly use, and which failure modes other people have already hit so that you are not the one discovering them in production. That list is short and it is genuinely obligatory, because an embeddings endpoint reaching end-of-life with a three-week re-indexing job behind it does not care whether you were paying attention. Most of what arrives alongside it — sentiment, positioning, who is thought to be ahead this month — is optional reading that presents itself as obligatory.

The default method for covering that ground is to follow everything and let volume stand in for judgement: subscribe to the newsletters, follow the practitioners, keep the tab open, and trust that anything important will float past eventually. That is the firehose, and it degrades in two directions at once. Attention is fixed at a few hours a week while the volume grows with the number of people writing about the field, so the fraction of it you actually read falls every quarter, and that fraction is chosen by whatever the social graph amplified rather than by what bears on your product. The 41 items are the small version of the failure: ninety minutes spent, six artefacts' worth of information obtained, and a false sense of corroboration acquired on top. Sustainably is the word doing the work in this topic's title, and it means an arrangement that costs the same few hours in year three as it did in year one and still catches the deprecation notice.

Two instruments carry that load, and this topic builds both. The first is a curated source list: a small written-down set of feeds, on the order of a dozen, where every entry was admitted by a stated rule rather than by having looked interesting on the day, and where the list has a fixed number of slots so that adding something obliges you to remove something. It governs intake, which is the decision about what reaches your desk at all — the decision the saved-links folder was making by default, and making badly. You meet it once a week, in an hour on the calendar where everything captured since the last one gets worked through in a single pass; chapter 7 covers how that hour is run and how you tell whether it is earning its place.

The second instrument answers a question the first cannot touch, since knowing that a model shipped and what it costs still leaves open whether it is any better at the thing your product actually does. A personal evaluation suite is a fixed set of your own test cases — inputs taken from your real traffic and your past incidents, each paired with what a correct answer has to contain — stored so that the same cases can be re-run against a model that did not exist when they were written. It is the stable ruler you hold up to every release, and you reach for it on the Wednesday after a Tuesday launch, and again on the Thursday when somebody senior asks whether you are switching. Chapters 2 through 6 build it, watch it fail, and repair it. The rest of this chapter stays on intake, since a list that admits everything guarantees the measurement work never gets the hours.

What the six were, and what the thirty-five added#

Sorting the 41 by provenance rather than by how interesting they looked is the whole move, and it is worth doing once by hand to see how lopsided the result is.

The two rows that carry information the vendor did not choose to give you are the two where someone ran the model themselves and showed their working. That is not a claim that vendors lie; model cards are generally accurate about what they measure. It is a claim about selection. A model card reports the benchmarks on which the model does well, at the prompt formats the lab tuned for, with the sampling parameters the lab prefers, and it omits the regression against the previous version on the one task your product happens to depend on, because nobody at the lab knows that task exists.

The 35 derived items are not worthless in aggregate — they are a reasonable signal of what people are excited about, which matters when your CEO is one of the people reading them. They are worth roughly one item's worth of attention rather than 35, and they are worth reading after the artefacts rather than before, since reading commentary first anchors you to a framing you then have to argue your way out of.

Three rules and a hard cap#

The reason lists grow is that admission is decided by interest at the moment of encounter, and everything is interesting at the moment of encounter. A rule fixes that by moving the decision to a calmer time and making it repeatable. Three criteria cover most of what a working engineer needs, and each of them is checkable by someone else, which is what stops the list from being a taste.

The first is provenance: the source is primary, or it is one hop away and shows its working. Release notes, model cards, API changelogs, pricing pages, papers, and repositories qualify directly. A person qualifies under the same rule when they publish the harness, the prompts and the raw outputs alongside the conclusion, because that is what makes the claim checkable rather than reportable.

The second is a checked track record. Before a person goes on the list, at least one of their past claims has to have been verified against something you control — you ran their prompt on your own data, or you reproduced their benchmark, or you shipped a change on their advice and watched what happened. One check is a low bar and it eliminates a surprising fraction of candidates, because most confident writing about models is never specific enough to be checked at all. Record what you checked and when, since the value of the record is that it survives your memory of being impressed.

The third is that the source bears on a decision you own. A lead engineer running a retrieval-heavy product in production owns decisions about serving cost, latency at their prompt lengths, structured-output reliability and long-context behaviour, and does not own decisions about training infrastructure, however interesting the training-infrastructure writing is. Sources that do not touch an owned decision go in a separate pile that has no obligation attached to it, which is the honest way to keep reading things for pleasure without pretending it is work.

Enforcement comes from a cap. A list with unlimited slots absorbs every new candidate, so the criteria never actually bind and nothing is ever removed. Give the list a fixed number of slots — twelve is a workable size for one person — and admission requires eviction, which forces the comparison that interest alone never triggers.

Twelve is not arbitrary, and the arithmetic behind it is worth doing for your own situation rather than accepting the number. The two major providers most teams depend on ship something worth knowing about roughly every three weeks, which is around 35 items a year between them. Deprecation and status feeds add perhaps a dozen. Two or three individuals who publish their own runs produce maybe 60 substantial pieces a year, of which a third touch anything you own. A curated paper feed adds another 40 abstracts, of which two or three get read properly. That totals somewhere near 150 items a year that deserve more than a glance, which is three a week, which fits in an hour with room left over. A list of thirty sources produces four hundred, which does not fit in an hour, and the thing that gives way is not the reading but the checking.

# sources.yaml — 12 slots, admission requires eviction
slots: 12
sources:
  - name: anthropic-release-notes
    kind: primary
    criterion: provenance
    owns_decision: [model_selection, api_migration]
    checked: n/a
    added: 2025-11-03

  - name: openai-changelog
    kind: primary
    criterion: provenance
    owns_decision: [model_selection, api_migration]
    checked: n/a
    added: 2025-11-03

  - name: simonwillison.net
    kind: one_hop
    criterion: track_record
    owns_decision: [model_selection, tooling]
    checked: 2026-01-14  # reran his structured-output test on our schema; matched
    added: 2025-11-03

  - name: vendor-status-and-deprecations
    kind: primary
    criterion: owned_decision
    owns_decision: [api_migration, incident_response]
    checked: n/a
    added: 2026-02-20

evicted:
  - name: ai-weekly-digest
    reason: derived; 9 of 10 items already on list via primary source
    evicted: 2026-02-20
    replaced_by: vendor-status-and-deprecations

A concrete admission shows how the three rules interact, since in practice a candidate rarely fails all of them. An engineer at another company began publishing detailed write-ups of long-context retrieval behaviour, with the prompts and the raw outputs included, which satisfies provenance comfortably. The track-record rule was not yet satisfied, so the candidate went into a probation pile rather than onto the list, and the check was scheduled: take one of the write-ups, run its central claim against the internal corpus, and record what happened. The claim was that recall collapses past a certain document count regardless of context window size, and reproducing it on 40 internal queries showed the same shape at a different breakpoint — the effect was real, the specific number was not transferable. That is a pass on the track-record rule, and the recorded caveat about the number is more useful eighteen months later than a memory of having been convinced.

The eviction that admission forced was the harder half. The twelve slots were full, and the weakest occupant was a well-written weekly digest that had been on the list for two years. Going back through eight weeks of it, nine of its ten most prominent items had already arrived through a vendor feed two or three days earlier, and the tenth was a piece of industry gossip. That comparison took twenty minutes and settled a question that had been generating low-grade guilt for a year.

The evicted block is doing more work than it looks like it is. Dropping a feed you have followed for two years feels like accepting a risk, and the feeling does not respond to arguments about attention budgets. It responds to a written reason, because a written reason converts the vague dread of missing something into a specific claim — nine of the last ten items reached me elsewhere first — that you can either believe or go and check. When something does eventually turn up that the dropped feed would have caught, the record tells you whether the eviction was wrong or whether you got unlucky once, and those deserve different responses.

What a curated list is not#

It is not a quality filter on writing. Some of the best-written commentary in the field is derived, and some primary sources are badly organised PDFs with the interesting number in a footnote. The criteria sort by provenance and relevance, and the reading skills from reading research are what you apply once you are inside the artefact — the list decides what reaches your desk, not what you conclude when it gets there.

It is not a strategy for being early. A curated list will routinely put you two days behind the social graph, because the model card goes up before anyone has run anything and the independent runs land later. Two days is almost never expensive. The situations where hours matter — a deprecation notice on an endpoint you depend on, a security advisory, a price change that alters your unit economics — are exactly the ones covered by vendor-primary sources with alerting attached, which is why those get slots even though nobody enjoys reading them.

It does not stop you missing things. What it changes is the shape of the misses. Following everything means missing things randomly, at a rate you cannot estimate, while feeling covered. Following twelve sources with written criteria means missing things in a known region — you will be late on developments in areas you do not own, and on people you have not checked — and being able to name that region is what makes recovery cheap, because a colleague's offhand mention of something you have not heard of maps immediately onto a gap you already knew you had.

The criteria also do not make sources trustworthy. Provenance tells you where a claim came from, and a primary source is not a neutral one; a model card is written by people with an incentive for the model to look good, and a release note omits the regressions nobody tested for. Preferring the model card to a thread about the model card gets you closer to the origin of the claim, without getting you closer to the truth of it.

That last gap is the one that actually matters when a decision is on the line. Having read the card, the changelog, the paper and two independent runs, you know a model exists, roughly what it costs, and how it performs on tasks chosen by other people. None of that answers whether it is better for your product, because your product's task is not on any benchmark, and nobody outside your company has ever run it. Answering that requires an instrument you own, built before the release rather than after it, and built so that a model announced next year can be dropped into it without a rewrite.