The stored chunk text usually outweighs the vectors themselves

The storage line in the quote from the previous chapter said "$9,400/month" and had been produced by multiplying 390 million chunks by 1536 dimensions by 4 bytes, getting 2.4 TB, and applying the storage price from the vector database's pricing page. The cluster that eventually served that index cost $19,900 a month, and the surprise was not in any single mistake. Every component that had been left out was individually small enough to sound like a rounding error, and there were six of them.

A vector index is closer to a materialised view than to a table. You do not store the thing you asked for; you store the thing you asked for, plus whatever structure makes it searchable, plus whatever you need to return alongside a hit so the caller does not have to make a second round trip, and each of those is a row width you chose implicitly rather than deliberately. Working out the per-chunk byte budget explicitly takes about ten minutes and it changes which optimisations look worth doing.

The structure that makes a vector index searchable deserves naming, because it is the second-largest line in that budget. Comparing a query against all 390 million stored vectors is arithmetic no product can afford on every request, so the store builds an approximate nearest neighbour index — ANN — which is a data structure that finds vectors close to the query while looking at only a few thousand of them, accepting that it will occasionally miss a true nearest neighbour in exchange for answering in milliseconds. Most stores build that structure as an HNSW graph, short for hierarchical navigable small world, in which every point keeps a list of M neighbours it links to at the base layer plus a thinner set of long-range links above, and a search walks those links downhill towards the query. The other structure in the budget is quantisation, which stores each of the vector's components in one byte instead of four so that the searchable copy shrinks fourfold, with the full-precision originals kept on disk so the handful of candidates a search returns can be rescored exactly before they are ranked.

The per-chunk byte budget, in two columns#

The distinction that matters most is not disk against total, but memory-resident against disk-resident, because RAM on a memory-optimised instance runs about $4.00 per GB-month while block storage runs about $0.08 per GB-month — a factor of fifty. Bytes that have to be in memory for the search to work are the expensive ones, and bytes you can leave on disk are nearly free by comparison.

Take a 390 million chunk index at 1536 dimensions, chunks averaging 410 billed tokens, which is roughly 1,600 bytes of UTF-8 text, with an HNSW graph at M = 32 and int8 scalar quantisation for the in-memory copy.

With text and metadata left on disk, the memory-resident footprint is 1,872 bytes per chunk, which is 730 GB, which at a replication factor of 3 — three complete copies of the index held on separate nodes, so that the service survives losing one — is 2,190 GB of RAM and $8,760 a month. The disk side comes to 10,515 bytes per chunk including slack, 4.1 TB, 12.3 TB across replicas, and $984 a month. Six times as many bytes on disk as in memory, and one-ninth of the cost.

Now move the payload into memory, which is the default in several stores and the configuration most teams end up with because it is what makes filtered search fast. The per-chunk memory figure becomes 4,092 bytes — 2,220 of which is text and metadata against 1,536 of vector — and the RAM bill goes to 4,788 GB and $19,152 a month. That is the $19,900 cluster. The chunk text really does outweigh the vector once the vector is quantised, and whether it lives in memory is a single boolean in a collection definition that nobody discussed.

def index_footprint(
    n_chunks: int,
    dim: int,
    quantised_bytes_per_dim: float = 1.0,   # int8
    hnsw_m: int = 32,
    text_bytes: int = 1_600,
    meta_bytes: int = 620,
    payload_in_memory: bool = False,
    replicas: int = 3,
) -> dict:
    graph = hnsw_m * 2 * 4 + 32            # layer-0 edges + upper layers + node overhead
    mem = dim * quantised_bytes_per_dim + graph + 48
    disk = dim * 4 + 48                    # float32 originals kept for rescoring
    if payload_in_memory:
        mem += text_bytes + meta_bytes
    else:
        disk += text_bytes + meta_bytes
    disk *= 1.25                           # segment slack, WAL, tombstones
    gb = 1e9
    return {
        "memory_gb": n_chunks * mem * replicas / gb,
        "disk_gb": n_chunks * disk * replicas / gb,
        "usd_month": (n_chunks * mem * replicas / gb) * 4.00
                     + (n_chunks * disk * replicas / gb) * 0.08,
    }

Two things in that function are worth arguing with rather than accepting. The replicas multiplier applies to memory and disk alike, so a decision made for availability triples the largest line in the bill, and it is often possible to run the replica set asymmetrically — full copies for the query path, a single copy for the batch analytics path that does not need five-nines. The 25% disk slack — partly filled segments, the write-ahead log that makes a write durable before it is merged into the index, and the tombstones that mark deleted vectors until a later merge reclaims their space — is a planning figure rather than a law; stores with aggressive segment merging spike well above it during compaction, which is a transient the cluster still has to be sized for.

Three things the table still leaves out are worth adding before the figure is quoted to anyone. Filter payload indexes are the largest: if queries filter by tenant, language and document type, each of those fields carries its own inverted or roaring-bitmap index — a compressed set of the point ids holding each value, which the search consults to know what it is allowed to visit — and on a 390 million point collection that typically adds 40 to 120 bytes per point per indexed field, and those indexes are memory-resident because filtering happens during graph traversal. Three filterable fields can therefore add as much memory as a fifth of the vector. Snapshots are the second: a daily snapshot retained for seven days is seven copies of the disk footprint unless the store does incremental snapshots, which turns 12.3 TB into something between 15 and 40 TB depending on churn. The third is the gap between provisioned and used, since you cannot run a memory-resident index at 98% of RAM — 25 to 30% headroom is normal for merge activity and query working set, so the 2,190 GB figure is a cluster of about 2,850 GB.

Why a third fewer dimensions is not a third off the bill#

A newer model at 1024 dimensions instead of 1536 removes a third of the vector bytes. What fraction of the bill that is depends entirely on how much of the bill is vector bytes.

In the first configuration the saving is 27%, close to the naive third because memory is mostly vector. In the second it is 12.5%, because two thirds of each resident chunk is text and metadata that a dimension change does not touch. The monthly numbers are $8,760 falling to $6,364, or $19,152 falling to $16,754. Moving the payload out of memory saves more than the model change does, costs nothing in API spend, and can be done this afternoon.

That comparison is the honest frame for any dimension-reduction argument. It is also the frame for the truncation techniques that have made dimension a tunable rather than a fixed property of a model: an embedding trained with a Matryoshka objective, which pushes the model to carry most of the meaning in the leading components so that a prefix of the vector stands on its own, can be cut to 512 dimensions at inference time with a measurable and often small recall loss, which means the dimension question is decoupled from the model question. Cutting 1536 to 512 in the payload-on-disk configuration takes 1,872 bytes to 848 and the RAM bill to $3,970 a month. Whether the recall loss is acceptable is a measurement, not an assumption, and it is the same measurement the last chapter of this topic is about.

Two indexes at once is 1.73x, not 0.67x#

The arithmetic that catches people out is the transition. Replacing a 1536-dimension index with a 1024-dimension one does eventually reduce the footprint to 0.73 of what it was, in the payload-on-disk configuration. Getting there requires a period in which both exist, because you cannot validate an index that has not been built and you cannot delete an index that is still serving traffic.

During that window the memory-resident footprint per chunk is 1,872 for the old index plus 1,360 for the new one, which is 3,232 bytes, which is 1.73 times the starting figure. In gigabytes that is 1,260 GB before replication and 3,781 GB after, against 2,190 GB today. The monthly cost during the window is $15,124 rather than $8,760, and the capacity that has to be provisioned is the peak rather than the average, because a cluster cannot be 73% larger on average and correctly sized at every instant.

There is a further transient on top of it. Building an HNSW graph needs working memory above the size of the finished structure — roughly 1.2 to 1.4 times, depending on the implementation and the build parallelism — so the true peak during the new index's construction is higher than 1.73x, though it is short-lived and can be smoothed by building in shards rather than all at once.

The levers on that peak are worth knowing before someone proposes buying 73% more cluster for a quarter. Building the new index at a replication factor of 1 and scaling to 3 only at cutover takes the window's total from 3,781 GB to 2,610 GB, a 31% reduction in peak, at the cost of having no redundancy on an index that is not serving traffic yet — which is usually an acceptable trade, since losing it means rebuilding rather than losing data. Building it in a separate cluster on cheaper capacity, or holding the new index's payload on disk while the old one holds it in memory, are variations on the same idea. Shard-at-a-time cutover, where a tenth of the corpus moves at a time, keeps the peak near 1.1x and stretches the window; it is only available if your routing layer can split traffic by shard, and it complicates the evaluation because the two models are never compared over the same corpus at the same moment.

What this arithmetic does not tell you#

It is a capacity model and not a latency model. Nothing in the byte budget says whether queries will be fast; that is set by the graph parameters, the number of segments, the filter selectivity and how much of the working set is actually resident, and an index that fits in memory with 5% headroom behaves very differently from one with 40%. Two indexes that both fit have the same storage bill and can differ by 4x in p99, the response time that only one query in a hundred exceeds.

On a managed vector service the whole calculation changes shape without changing its conclusion. Those services bill per stored vector per month with a dimension band and a payload allowance, so the levers look different — you are choosing a tier rather than sizing RAM — but the underlying quantities are the same, and the reason the tier jumps when you cross 1024 dimensions or exceed the included payload bytes is precisely the arithmetic above. Being able to reproduce a vendor's price from the byte budget is what tells you whether a quoted tier is generous or whether you are paying twice for text you also keep in Postgres.

It also does not price the CPU. Building an HNSW graph over 390 million vectors at ef_construction = 200, the number of candidate neighbours the builder weighs before choosing the links it keeps for each point, is on the order of 3,000 to 5,000 core-hours, which is a real line item on a reindex and appears nowhere in a byte count. And it says nothing about whether the quantisation that made the memory figure work is safe on your corpus, since int8 scalar quantisation typically costs 1 to 3 points of recall@10 — the share of a query's true ten nearest chunks that the index actually returns — and occasionally much more when the embedding distribution is unusually anisotropic, with its variance concentrated in a few directions rather than spread evenly.

With cost per chunk and cost per byte both pinned down, the remaining question about a large embedding job is not what it costs but whether it can finish. A provider that will sell you 160 billion tokens for $16,000 will not sell you them all at once, and the rate at which it releases them decides your schedule.