Dimensionality is a storage bill before it is a quality decision

An embedding model is a program that takes a passage of text and returns a fixed-length list of numbers. The length of that list is the model's dimension, and it is a property of the model rather than of the text: a one-word passage and a page-long passage come back as lists of exactly the same length. Dimension is usually argued about as a quality knob, on the reasoning that a longer list has more room to record what a passage is about. That is a defensible reading of what the number does, and it is the second thing the number does.

The first is arithmetic. Width appears as a multiplier in three separate places — the bytes a store holds, the bytes a single query moves, and the size of the machines that have to be big enough for the first — and each of those multiplications happens before any question of retrieval quality is reached. A chunk, throughout, is a passage-sized piece of a document, a few hundred tokens long, that gets embedded once and stored as one vector. The unit of the bill is therefore the vector, and the bill is the number of vectors times what one of them costs.

The store is a rectangle#

Work out one vector first, because everything after it is that figure scaled. A vector from a 1024-dimension model is an array of 1024 numbers, and each of those numbers is stored as a float32 — a single-precision floating-point value, which occupies four bytes of memory whatever value it holds. So one vector costs its component count multiplied by the bytes per component: 1024 times four bytes, which is 4 KB. Nothing about the passage that produced it changes that figure.

Every vector in the store has that same width, which is what makes the total easy. Laid out in memory the store is a rectangle: one row per chunk, and every row exactly 4 KB wide. The size of a rectangle is its height times its width, so the raw storage is the number of chunks times the bytes per vector. Ten million chunks, 1024 dimensions each, four bytes per component gives 40.96 GB of raw vector data, and the reason it is a single multiplication rather than an estimate is that no row is wider than any other.

Two consequences follow directly from that shape. The first is that the raw figure does not depend on the documents in any way — not their language, not their length, not how much they overlap — only on how many chunks there are and how wide the model is. The second is that the relationship to width is exactly linear with no threshold in it: halving the dimension halves the row, and halving the row halves the rectangle. That linearity is the whole reason dimension is worth arguing about in a budget meeting, and not every term in the real bill behaves that way.

What the raw figure leaves out#

The 40.96 GB is the number people quote as if it were the answer, and it is closer to a third of what actually gets provisioned. The index structure, the payload, and the replicas that keep the service up are all multipliers on top of it, and each one is applied to the raw figure rather than absorbed by it.

An HNSW index — hierarchical navigable small world, the graph structure most vector databases use to find near neighbours without comparing the query against every stored vector — stores a neighbour list per vector on every layer the vector appears in. With M=16, the base layer keeps up to 32 neighbours and the upper layers add roughly another sixteen amortised across all nodes, so at four bytes per neighbour identifier the graph costs about 190 bytes per vector, or 1.9 GB across ten million. Raising M to 32 for better recall on a hard corpus roughly doubles that.

Three steps in that sentence are worth separating, because the graph row is the one that will not move later. The base layer is built with double the budget of the upper ones, which is why M=16 allows up to 32 neighbours there. Every vector sits on the base layer and only a shrinking fraction is promoted to each layer above, so the upper layers contribute a small constant per vector rather than a multiple of the layer count. Adding the two neighbour counts and multiplying by the four bytes an identifier takes produces the 190 bytes quoted — and what is being counted there is identifiers, integers that name other vectors, not coordinates. A neighbour list is the same size whether the vectors it points at have 1024 components or 64.

Payload is usually the larger surprise: if the chunk text lives in the same store, 400 tokens of English is about 1.6 KB, which is 16 GB — often more than the vectors, and the reason many teams keep text in Postgres and only identifiers in the vector store. That figure comes from the tokens rather than from the model, since 1.6 KB is what 400 tokens works out at around four bytes of English per token. It scales with how much text a chunk holds and is completely indifferent to the width of the vector beside it, so it is the one term in the bill that dimension reduction cannot touch.

Then the whole thing is multiplied by availability. A single node holding the index is a single point of failure and a single point of restart latency, so production usually means two or three replicas, and if the store shards for write throughput the shards each carry their own overhead. A replica exists to serve queries when another node stops, which means it has to hold everything the stopped node held; replication therefore multiplies the footprint rather than dividing it. Working from the 40.96 GB base, an index built with M=16, text held externally, and three replicas comes to roughly 129 GB of resident memory. Memory-optimised instances at that size run around $0.80 per hour each for 128 GiB, so three of them is about $1,750 per month, which is the number an infrastructure review is actually asking for.

The memory column closes on itself exactly, and that is worth checking rather than assuming, because it decides whether anything unnamed is hiding in the total. The two component rows add to the subtotal in both columns, and the subtotal multiplied by three gives the three-replica row in both columns. One detail in that second step looks like an error and is not: three times the printed 42.9 is not 128.6, and the row is right because the multiplication was done on the unrounded subtotal before either figure was rounded for display. The same holds in the 256-dimension column. Since the parts reproduce the whole with no residual, there is no unitemised overhead sitting in the memory figures, which licenses a strong statement about them: the only way to move that column is to move one of the two named rows, and only one of the two responds to dimension at all.

The price row does not close in the same way, and the gap has a name. Dividing the monthly figure by three and by the hours in a month returns the quoted hourly rate, so the arithmetic from instance to bill is sound. What does not follow from the memory column is the instance size: each replica has to hold 42.9 GB and the instance being priced offers 128 GiB, a gibibyte being the binary unit and therefore slightly larger than the gigabyte the table's other rows are counted in. The bill is set by provisioned capacity rather than by resident bytes, and the difference between the two is headroom — for index builds, for query working memory, for growth past the projection. It is real money and it is unitemised, so quote the price as what it is rather than deriving it from the footprint and expecting them to agree.

The row that does not change is worth noticing, because it explains where dimension reduction stops helping. The graph cost is per vector rather than per float, so at 64 dimensions the graph would be about 40% of the total and halving the dimension again would save far less than the arithmetic suggests. Run that check with the numbers: at 64 dimensions the rectangle is ten million rows of 64 components at four bytes, which the table below gives as 2.6 GB, while the graph is still 1.9 GB. The graph is now a comparable size to the thing it indexes, so a further halving takes half of the smaller term and none of the larger one.

The other cost is bandwidth on every query#

Storage is a monthly bill and latency is a per-request one. A graph search does not scan every stored vector, but it does compute a distance for every candidate it visits, and each of those distances requires reading a full vector from memory at a location the prefetcher cannot predict. The reason the prefetcher is helpless is structural: the search only learns where the next candidate lives after it has read the current candidate's neighbour list, so the addresses arrive one at a time and cannot be guessed ahead. Every visit is therefore a fresh random access, and the bytes it moves are the full width of one vector.

That gives the per-query traffic as a multiplication of the same shape as the storage one, with the number of visits standing in for the number of chunks. A search with ef_search=64 — the size of the candidate list the search keeps while it traverses, where a larger list means more nodes visited and better recall — on a ten-million-vector index typically visits somewhere between 300 and 600 nodes across all layers. At 1024 float32 dimensions each vector is 4 KB, so a single query pulls one and a half to two and a half megabytes of effectively random memory traffic before any of the SIMD arithmetic — the wide instructions that multiply many floats at once — happens. At 256 dimensions the same search touches the same number of nodes and pulls a quarter of the bytes.

That relationship is close enough to linear in dimension that you can predict it, and it compounds with concurrency in a way that a single-threaded benchmark hides. One query at a time on an idle box shows a few hundred microseconds of difference and looks like it does not matter. At 200 queries per second across sixteen cores, the 1024-dimension index is asking for around 400 MB/s of random reads while competing for the same last-level cache, and p95 latency — the figure only one query in twenty exceeds — separates from p50, the median, much earlier than the low-dimension index does.

Those last two figures are independent statements that have to agree, and they do: the middle of the per-query range multiplied by the query rate is the 400 MB/s quoted, which means the aggregate figure and the per-query one are the same measurement seen twice. The cache pressure follows from the same rectangle. A few hundred vectors at 4 KB each is already a substantial fraction of a last-level cache, and sixteen cores are each doing that at once, so the lines one query pulls in are evicted before the next query can reuse them. That is why the penalty shows up in the tail rather than in the median.

Recall@5 in the table below is the share of queries whose judged-relevant chunk appears in the top five results, measured against a golden set — a fixed list of queries, each with the chunks a human has judged relevant to it.

Read that table as differences between adjacent rows rather than as five levels, because the decision is always about one step. Going from 512 to 256 halves the storage column and costs a little over one point of recall. Going from 128 to 64 halves the storage column again and costs more than five times as much recall as that step did. The bill responds to every halving identically, by construction, since it is a rectangle; recall does not, and the ratio between those two gaps is the entire trade.

The latency columns bend the same way and for the reason the graph row gave earlier. The first two steps take large bites out of p95, and the last step takes almost nothing, because part of the per-query work is visiting nodes and following pointers rather than reading floats, and that part does not shrink when the vectors do. So at the bottom of the table each halving buys less latency and less footprint in absolute terms while costing steadily more recall. Those five rows come from one model, and that is the only way the table means anything.

Varying dimension without varying the model#

The tempting way to produce a table like the one above is to take a family that publishes several sizes — a small, a base and a large — and measure each. That measures something real and it does not measure dimension, because the small model is also shallower, has fewer parameters, and was trained with a different compute budget. A drop from the large to the small conflates capacity with width and cannot be attributed to either.

Two clean methods exist. The first is Matryoshka representation learning, or MRL, where a model is trained so that the first n dimensions of its output are themselves a usable embedding, with the training loss applied at several truncation lengths. Models built this way state it plainly on the card and list the supported lengths — nomic-embed-text-v1.5 supports 768 down to 64, and OpenAI's text-embedding-3 family exposes it as a dimensions request parameter down from 3072. Truncating such a model gives you one encoder measured at several widths, which is exactly the experiment.

import numpy as np

def truncate(vectors: np.ndarray, dims: int) -> np.ndarray:
    """Matryoshka truncation: keep the leading dims, then renormalise."""
    head = vectors[:, :dims]
    return head / np.linalg.norm(head, axis=1, keepdims=True)


full = np.load("chunks_768.npy")          # (10_000_000, 768) float32
for d in (768, 512, 256, 128, 64):
    small = truncate(full, d)
    np.save(f"chunks_{d}.npy", small.astype(np.float32))

The renormalisation is not optional and is the most common way this goes wrong. Dropping the tail of a unit vector leaves a vector of norm less than one, and the norms differ per document — a document whose information is concentrated early keeps a norm near 0.95 while another falls to 0.7. Cosine similarity computed as a raw dot product on those vectors then systematically favours documents whose meaning happened to load onto the early dimensions, which produces a ranking that looks subtly, inexplicably wrong rather than obviously broken.

The chain that gets a team there is short enough to follow, and each link is something a reasonable person did on purpose. Cosine similarity is a dot product divided by the two vectors' lengths. Encoders normally return vectors of length one, so that division multiplies by one and vector stores skip it as a pure waste of instructions. Truncation quietly withdraws the condition the optimisation depended on, since a shortened vector no longer has length one, and the amount by which it falls short is a property of the individual document. Skipping the division is then no longer free, and it tilts every comparison in favour of the documents that kept the most length. Dividing each truncated vector by its own new length, which is what np.linalg.norm in the code above computes, puts the condition back.

The second method is the hosted equivalent, where the provider does the truncation and normalisation:

resp = client.embeddings.create(
    model="text-embedding-3-large",
    input=chunks,
    dimensions=256,          # 3072 by default; MRL-trained truncation
)

Truncating a model that was not trained for it destroys far more than the arithmetic suggests. On a conventional 1024-dimension encoder, cutting to 256 dimensions moved recall@5 from 0.781 to 0.594 in one measurement, against 0.766 for an MRL model at the same width, because a conventional encoder distributes information across all dimensions with no reason to concentrate it early. The model card is where you check, and absence of a claim means no.

Those two results sitting side by side are the point at which a very common belief breaks, and it is worth naming because it is tempting rather than careless. The belief is that a benchmark score justifies the wider model: a 1024-dimension encoder wins a head-to-head, its extra width is taken to be what won it, and cutting that width is expected to give the win back. The score is a measurement of the encoder at its full width and says nothing about what its leading 256 components do on their own. Front-loading the discriminative information into the early dimensions is a property a provider chose to train in, and it is not a property embeddings have by default — which is exactly what the drop to 0.594 measures.

The shape of the recall column is the part worth internalising, because it is not linear and it is fairly consistent across MRL models. Cutting the trained width in half costs almost nothing — 768 to 512 to 256 loses under two points in total — and then the curve turns sharply, with 128 costing five points and 64 costing thirteen. Those last two figures are counted from the trained width rather than from the row above, which is the reading that makes them consistent with the first clause. The reason is that the training objective packs the most discriminative directions into the leading dimensions, so the tail is carrying refinements rather than substance until you cut deep enough to start removing substance. Where the knee sits varies by corpus: a corpus with many near-duplicate documents needs more dimensions to separate them and its knee arrives earlier, which is another reason to measure on your own material rather than trusting the published truncation curves.

Compression is a second, separate lever#

Reducing dimension is one way to shrink a vector; reducing the precision of each component is another, and the two are independent. One shortens the row by removing components and the other narrows every component in place, so a store can do both and the savings multiply. Scalar quantisation stores each dimension as an int8 rather than a float32, which is a flat four-fold saving and typically costs half a point to a point of recall on a normalised encoder. Binary quantisation stores one bit per dimension, a thirty-two-fold saving, and on its own it costs a great deal — but it is usually deployed with rescoring, where the binary index retrieves an oversampled candidate list and those candidates are re-ranked with full-precision vectors fetched from disk. With 4× oversampling, that recovers most of what the binarisation lost while keeping the resident index tiny.

The bytes column is the same rectangle with a different row width, and it is worth doing once to see that the savings are not estimates. A 768-dimension float32 vector is 768 components at four bytes, the 3,072 in the first row. An int8 holds one byte, so the row width becomes the component count itself. A binary code holds one bit per component, so the row width is the component count divided by the bits in a byte, which is the 96 in the third row and exactly a thirty-second of the first. Multiplying each row width by ten million gives the column beside it, which is why the storage saving from a representation change is knowable before anything is measured while the recall cost is not.

Rescoring changes the latency profile rather than the storage one: the graph search gets much faster because Hamming distance, the count of bit positions where two of those 96-byte codes differ, is trivial to compute, and then a disk or object-store read is added for each of the oversampled candidates. On local NVMe, a solid-state disk attached to the machine itself, that is a millisecond or two; on network storage it can be twenty, which reverses the whole trade.

Both levers are recall-for-cost trades and should be written down as such. A proposal that says "we moved to 256 dimensions and saved 70% of the memory bill" without the recall column is describing half a transaction, and the half it omits is the one the retrieval team will discover in three months.

Width compared across models#

A larger vector is not a better vector, and the cross-model version of that claim is where the confusion starts. A 1536-dimension model losing to a 768-dimension model on your corpus is completely ordinary and says nothing about width, since the two were trained on different data with different objectives, and the head-to-head of the previous chapter — two encoders over one judged query set with everything else held still — is what settles which one retrieves better. Dimension only becomes a quality variable inside a single model, and then it behaves as the tables above show: nearly free down to about a third of the trained width, then increasingly expensive.

Dimension also does not matter at small scale, and treating it as a headline decision on a 200,000-chunk corpus wastes effort. Two hundred thousand vectors at 1024 dimensions is 0.8 GB, which fits comfortably in any instance you would run, and the p95 difference between widths at that size is well inside the noise of the rest of the request. The arithmetic earns its place at the corpus size you are projecting rather than the one you have, which is why the measurement should be run against a synthetically padded index at the projected count.

Padding an index up to the projected count#

Measuring at a corpus size you do not yet have is straightforward and is usually skipped because it sounds like it needs the documents. It does not: an index's memory footprint and search latency depend on the number of vectors and their width, so padding the real 200,000 chunks with 9.8 million synthetic vectors drawn from the same distribution — sampling real vectors and adding small Gaussian noise, which preserves the local density that a graph index is sensitive to — produces an index whose latency and footprint are representative. Recall against your judged documents will be pessimistic in that index because the synthetic neighbours are unnaturally close, so the three columns come from two runs: footprint and latency from the padded index, recall from the real one. Purely random vectors are the wrong padding, since they spread out uniformly, leave the graph unnaturally easy to traverse, and produce latency numbers that flatter every dimension equally.

That split between the two runs follows from which quantities the padding can and cannot fake. Footprint is a count and a width, both of which the padded index has honestly. Latency depends on how many nodes a search visits, which depends on how tightly packed the neighbourhood around a query is, and noisy copies of real vectors reproduce that packing closely enough to trust. Recall depends on which document comes back first, and there the synthetic vectors are active liars — each one sits beside a real vector by construction and competes with it for the top five. So the padded index answers two of the three questions and must not be asked the third.

The scale at which these levers are worth pulling is also the scale at which the rescoring trade above can invert. Binary codes with disk rescoring are attractive precisely when the resident footprint is the constraint, and where the rescoring reads come off network storage a team can save the storage bill and lose more p95 than the wider float32 index ever cost them.

Dimension is a purchase before it is a quality decision, and the purchase is made in three currencies at once: resident bytes, which are a rectangle of vectors by components by bytes-per-component; memory bandwidth, which is that same row width multiplied by however many nodes a search visits; and instance hours, which are the first currency rounded up to whatever the provider sells. Only the first two are linear in width, the third is bought in lumps, and the graph structure that makes the search fast sits underneath all of them at a constant cost per vector that no amount of narrowing will remove. Quality enters after that, and it enters as a curve that is nearly flat down to about a third of the trained width and steep below it.

One property of these levers matters for what comes next. Truncation and quantisation are transformations of vectors you already have, so shrinking a 768-dimension index to 256 dimensions is a batch job over stored data that never calls the encoder — you can do it overnight, keep the originals, and reverse it. Changing which model produced those vectors offers nothing of the kind, because the old vectors carry no information the new model can use, and that asymmetry is what makes a model swap a different class of operation.