Model landscape literacy

Read these first

• Base vs instruct vs reasoning vs multimodal variants

What you will know

• Build a comparison table for three candidate models across quality, latency, context, modality, deployment, and price

  Include token direction, rate limits, and hosting assumptions; list prices alone hide batching, minimums, and output-token costs.

• Read a model card and extract the facts that matter for a decision

  Record caveats such as evaluation data overlap, unsupported languages, safety refusals, licence restrictions, and missing details—not just the headline context window.

• Run a task-specific bake-off that compares quality, latency, cost, and failure modes across the shortlisted models

  Use a fixed representative set and repeat enough requests to expose variance; a public benchmark win may disappear on private inputs or under production concurrency.

• Verify whether an open-weight model can legally and operationally be deployed for the intended use

  Accessible weights do not guarantee commercial rights, permissive training-data terms, safe redistribution, or a practical serving stack.

• Shortlist three models for a task and say what would settle the choice

  The deciding test should name its data, success threshold, and operating budget so the choice can be resolved rather than argued from leaderboard rank.

Chapters

1. 1

   Open-weight models

   An open-weight model is one whose trained parameters can be downloaded, and the term guarantees nothing past that. Commercial use may be restricted, redistribution forbidden, training-data provenance undocumented, and no practical serving stack may exist for the hardware you can rent. This chapter separates the four gates a candidate has to pass — licence grant, acceptable-use policy, training-data terms and serving feasibility — and shows the half-day check that runs all four before an open-weight candidate is allowed into a comparison.

   Read chapter
2. 2

   Price per million tokens

   A single price per million tokens is a weighted average whose weights come from your traffic, not from the provider. It conceals the input/output split, where output usually costs three to five times more, and it sits on top of batch discounts, context tiers, rate limits, cached-input rates and per-modality schedules. It also explains why one open-weight model appears twice on a marketplace at nearly double the price. This chapter builds the comparison table those mechanisms require and the one derived cell worth comparing directly.

   Read chapter
3. 3

   Self-hosting break-even volume

   Self-hosted inference is sold per GPU-hour, so comparing it with a per-token price means one division performed twice: the hourly rate of the instance you would rent, over the tokens per second that model actually achieves on it at your batch size and sequence lengths, over a utilisation factor that admits how much of the day the GPU sits idle. The result is a break-even request volume that moves with quantisation, batch size, context length and traffic shape — and three cost lines that sit outside it entirely.

   Read chapter
4. 4

   Benchmark scores and model cards

   A published benchmark score has to survive three separate questions before it informs a decision: whether it was produced under conditions comparable to anyone else's, whether the model had already seen the test items, and whether the benchmark measures anything you are buying the model to do. This chapter works through protocol variance, contamination detection, the facts a model card omits, and construct validity — and settles on the one job published scores do well, which is eliminating candidates rather than choosing one.

   Read chapter
5. 5

   The evaluation specification

   A shortlist argued about in leaderboard positions never terminates, because no statement exists which, if true, would end it. An evaluation specification — test data, absolute thresholds and an operating budget, written down and timestamped before the first run — converts the shortlist into a decision procedure. This chapter covers how to construct each part, why absolute thresholds make 'none of them' a legitimate outcome, and what pre-commitment does not protect you from.

   Read chapter
6. 6

   The bake-off

   An offline score of 0.74 against 0.72 may not be two different numbers, and it says nothing about what the service will meet in production. A bake-off tests the shortlist on four dimensions an offline evaluation cannot reach: whether the margin survives resampling, how each candidate behaves at real peak concurrency, which failure modes it produces rather than how many errors, and what its terms would oblige you to. This chapter runs all four and shows where the offline winner loses.

   Read chapter
7. 7

   The operational commitment of self-hosting

   \"Self-hosting is cheaper at scale and avoids vendor lock-in\" is two claims joined by 'and'. The first is a quantity computed in chapter three and often unlocated. The second is a category error: self-hosting moves a dependency from an organisation you pay to a capability you must staff, and a platform is harder to exit than an API contract. This chapter prices the obligation, works through what 200 enterprise customers change legally, and records a default with the triggers that would reverse it.

   Read chapter