The same vector length comes back for five words or five hundred

Support search of the kind this topic is about works by comparison rather than by keyword match. A support corpus — the whole set of help articles, runbooks and reference pages a company has written — is split into chunks, passages of a few hundred words each, because a whole document is too coarse a unit to hand back as an answer, and every chunk is passed through a model that converts text into a fixed-length list of numbers. A user's question goes through the same model when it arrives, and the service answers by finding the stored lists that are numerically closest to the question's. That indirection exists because the words rarely line up: someone typing "card declined" needs the page headed "Failed payment authorisations", and no amount of string matching gets from one phrase to the other, while the two pieces of text land close together once they are numbers.

Each of those lists is an embedding — a fixed-length vector that a model produces from a piece of text, positioned so that texts with similar meaning end up near one another, which is what is meant by calling the arrangement a semantic space. They are dense vectors, in that nearly every coordinate is a small non-zero number carrying part of the signal, as opposed to the mostly-zero keyword-count vectors a classical search index stores. A backend engineer meets them behind documentation and product search, "related articles" panels, near-duplicate detection, recommendation, and the retrieval step that gathers context for a chatbot before it answers. What holds them is a vector index: a table of these vectors with a structure over it for finding near neighbours quickly.

The question this chapter settles is which part of that fixed-length list the input controls. The answer is none of it. Nothing about that is obvious from the outside: a function that takes a string and returns a list of floats looks like it ought to return more floats for more string, in the way that a hash of a large file takes longer than a hash of a small one and a compressed archive of a book is bigger than a compressed archive of a paragraph. Which of those two intuitions is the right analogy turns out to matter for sizing a database, for setting a chunk length, and for what you write in a schema review, so it is worth settling with two printed numbers rather than with reasoning.

Printing the thing#

import numpy as np
from openai import OpenAI

client = OpenAI()
MODEL = "text-embedding-3-small"

short = "The API returns 200 on success."
long = open("runbook-restart-procedure.md").read()  # 2,140 words

resp = client.embeddings.create(model=MODEL, input=[short, long])
a, b = (np.array(d.embedding) for d in resp.data)

print(len(a), len(b))
print(a[:5].round(4))
print(round(float(np.linalg.norm(a)), 6), round(float(np.linalg.norm(b)), 6))
print(resp.usage.prompt_tokens)

1536 1536
[-0.0121  0.0338 -0.0075  0.0261  0.0044]
1.0 1.0
2993

Seven words and 2,140 words both came back as 1,536 float32 values, which is 6,144 bytes each on the wire and in the index, before whatever overhead the store adds. The prompt token count differs by a factor of four hundred; the output size does not differ at all. Storage for the vectors is a function of how many chunks you have, and the per-chunk cost is a constant you can read off the model card before you write a line of code.

The last printed line is worth a second look, because it is the two facts side by side in one number. resp.usage.prompt_tokens is the total for the call, and the call had two inputs in it; the table below records those two inputs at 8 tokens and 2,985 tokens, which add to exactly the 2,993 that was printed. Billing is a sum over inputs, so it grows with every token in every one of them. Storage is a count of inputs times a constant, so it grows only when the count does. Both quantities came out of the same HTTP request, and they obey different laws.

Both vectors also came back with an L2 norm of exactly 1.0, which is that provider's choice rather than a general property — chapter 5 of the vector arithmetic topic dealt with which sources hand you unit-norm output and which do not, and this one happens to.

The useful way to read that table is not down the columns but across the gaps between rows. Between the first two rows the token count multiplies by several hundred and the byte count does not move. Between the second and the fourth rows the token count does not move at all and the byte count doubles, because the model changed. Between the second and the fifth the token count falls and the byte count falls too — but the arrow of causation still runs through the model column, since a different encoder both truncated the input and produced a narrower vector. No cell in the table contains the finding, which is that the byte column is a function of the model column alone: it is the vector length multiplied by four, because a float32 is four bytes, and 1536 × 4 = 6,144, 3072 × 4 = 12,288, and 384 × 4 = 1,536 with no residual anywhere.

That last row hides a coincidence worth naming before it bites, because these two columns of the table now share a numeral. The 1,536 in the third column of row one is a count of floats; the 1,536 in the fourth column of row five is a count of bytes, and it belongs to a 384-dimensional model. Anyone matching figures across a sizing spreadsheet by eye will eventually align those two, and the multiplication by four is the only thing that tells them apart.

Every row in that table varies the input and the length stays put; the length only moves when the model does. That is what dimensionality refers to for an embedding — the size of the output space the model was built with, fixed at training time, the same for every input it will ever see. Choosing 384 against 1536 against 3072 is a decision you make once when you pick a model, and it propagates into your column definition, your index memory footprint and the parameters of your approximate nearest-neighbour (ANN) index — the structure that finds close vectors without comparing a query against every row — which is why it belongs in a schema review rather than in a chunking discussion.

The scale of that decision is easy to work out because it is arithmetic. A corpus of one million chunks at 1,536 dimensions occupies 6.1 GB of raw float32 vectors, which is one million multiplied by the 6,144 bytes each vector takes, and a graph index such as HNSW — Hierarchical Navigable Small World, the most common ANN structure, which links each vector to a few near neighbours and walks the links at query time — typically adds somewhere between a half and a full multiple of that for its edge lists, so the working set lands around 9 to 12 GB and needs to fit in memory if you want single-digit-millisecond queries. Those two bounds are the raw figure plus half of itself, and the raw figure doubled; nothing else is in them. The same corpus at 384 dimensions is 1.5 GB raw, which fits comfortably on a machine a quarter of the size, and it is a quarter for the plainest possible reason: 384 is a quarter of 1536, and every byte in the estimate is proportional to the dimension. Both the distance computation at query time and the bytes moved from memory scale linearly with the dimension count, so the choice shows up in the latency budget as well as the bill.

Some providers let you have part of that back without changing models. The text-embedding-3 family was trained with a nested objective that keeps the most useful information in the leading coordinates, so truncating a 3,072-dimensional vector to its first 256 values and renormalising costs a few points of benchmark average rather than collapsing it, which is a meaningfully different proposition from lopping the tail off an ordinary embedding. That property belongs to specific models and is documented per model; assuming it holds generally is how people end up with an index full of vectors that have quietly lost most of their signal.

Where the fixed length comes from#

The constant width is not a convention the API imposes on its output. It falls out of the architecture, in a chain of four steps that is worth following once, because every later chapter in this topic leans on the last of them.

The transformer inside the model, the neural network that reads the whole sequence at once, does not produce one vector. It produces one per token: the input is a sequence, each position gets a representation, and for a 2,985-token input that is a 2,985 × 1,536 matrix of activations at the final layer. The width of that matrix is 1,536 because the width of every layer inside the model is 1,536, which was chosen when the model was designed and is baked into the shape of its weight matrices — it cannot vary with the input any more than the number of columns in a database table can vary per row. The height of the matrix is the only part the input controls. A single vector comes out because something collapses the height away, and once the height is gone the only number left is the width, which never depended on the text. With a hosted API that collapsing step happens on the other side of the HTTP boundary. Run a local model and it is visible in three lines:

import torch
from transformers import AutoTokenizer, AutoModel

name = "sentence-transformers/all-MiniLM-L6-v2"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModel.from_pretrained(name)

batch = tok(["The API returns 200 on success."], return_tensors="pt")
out = model(**batch).last_hidden_state
print(out.shape)                      # torch.Size([1, 10, 384])

mask = batch["attention_mask"].unsqueeze(-1).float()
pooled = (out * mask).sum(1) / mask.sum(1)
print(pooled.shape)                   # torch.Size([1, 384])

Ten tokens went in and ten 384-dimensional vectors came back from the encoder; the mean over the sequence axis, weighted by the attention mask so padding contributes nothing, produces the one 384-dimensional vector that gets stored. The middle axis of the printed shape is the one that disappears, and reading the two shapes against each other shows the whole mechanism: [1, 10, 384] becomes [1, 384], so the axis whose size was the sentence's length is summed away and the axis whose size was the model's width survives untouched. A sentence of a hundred tokens would print a hundred in the middle slot at the same line and the same [1, 384] at the next. Pooling is the name for that collapse, and the common variants are mean pooling as shown here, taking the vector at a designated [CLS] position, and taking the last token's vector, which is what most decoder-based embedding models do. Which one a model uses is a property of how it was trained, and using the wrong one — mean-pooling a model trained with CLS pooling, for instance — produces vectors that still have the right shape and still cosine against each other perfectly happily while retrieving noticeably worse.

Notice what the mean is being taken over. Division by mask.sum(1) is division by the number of real tokens, so each token position contributes a weight of one over the sequence length. The consequence of pooling is that information from many tokens is being averaged into one point, and a longer chunk gives each of its tokens a smaller share. A 200-token chunk about a single subject and a 200-token chunk that changes subject three times both yield one point, and the second one lands somewhere in between its three subjects rather than near any of them.

Pooling also explains why the output length has nothing to do with the input length while still depending on the input in a way that costs money. The encoder does real work per token — attention is quadratic in sequence length within a chunk, and the provider bills on tokens in rather than floats out — so a long document is expensive to embed and cheap to store, which is the reverse of the intuition a token-based sizing estimate is built on. At $0.02 per million tokens for text-embedding-3-small, embedding a corpus of four million 350-token chunks costs about $28 in tokens and produces 24.6 GB of vectors. Both halves of that are one multiplication. Four million chunks of three hundred and fifty tokens is fourteen hundred million tokens, and at two hundredths of a dollar for each million that is twenty-eight dollars. Four million chunks at 6,144 bytes each is a shade under twenty-five billion bytes, which is the 24.6 GB, and it is exactly four times the one-million-chunk figure from earlier because nothing in it depends on the tokens at all. The second of those two numbers is the one that will shape the architecture, and it is the one a per-token estimate cannot produce.

Measuring "close" instead of asserting it#

The claim that makes embeddings useful is that a paraphrase lands near its original. That claim is testable, and reporting it as a number costs one extra line:

sentences = [
    "The API returns 200 on success.",                        # base
    "A successful call to the API responds with status 200.", # paraphrase
    "The API returns 500 when the upstream times out.",       # same topic
    "The kitchen closes at ten on weeknights.",               # unrelated
]
vecs = np.array([d.embedding for d in
                 client.embeddings.create(model=MODEL, input=sentences).data])
sims = vecs[1:] @ vecs[0]      # unit norm, so dot == cosine
print(sims.round(3))           # [0.887 0.712 0.041]

Cosine 0.887 for the paraphrase, 0.712 for a sentence about the same API with a different status code, and 0.041 for the sentence about the kitchen. Those three numbers are worth more than the sentence "the paraphrase is close", because 0.887 on its own is uninterpretable and the spread between the three is not. Read them as two gaps rather than three levels. The step from the unrelated sentence up to the same-topic one is most of the distance available, and it is the model reporting that these two texts are about the same system at all. The step from the same-topic sentence up to the paraphrase is much smaller, and it is the only part that reports paraphrase specifically. Any later argument about whether a score is high enough is an argument about that second, narrower gap, which is why a threshold picked by eye from the first one will be far too generous.

The vector arithmetic topic made the case that a threshold is a fact about one model on one dataset; the same applies to any single score you quote, so a similarity claim needs the model name attached to be reproducible at all.

Rank carries more information than raw score when what you are actually testing is retrieval. Embed the base sentence as a query against a pool of real chunks and ask where the paraphrase lands:

pool = load_chunks("support-corpus.jsonl")          # 4,812 chunks
pool_vecs = embed_all(pool)                         # (4812, 1536)
pool_vecs = np.vstack([pool_vecs, vecs[1]])         # append the paraphrase

scores = pool_vecs @ vecs[0]
order = np.argsort(-scores)
print(int(np.where(order == len(pool))[0][0]))      # 0  -> rank 1
print(scores[order[:3]].round(3))                   # [0.887 0.804 0.79 ]

Rank 1 out of 4,813, with the second-place chunk at 0.804. The margin between first and second is small enough that a slightly different paraphrase, or a corpus with one more near-duplicate in it, would reorder them, and that fragility is invisible if you only look at the top score. Reporting both the score and the rank, against a named model and a named pool, is the difference between a demo and a measurement.

What the fixed length does not buy you#

A constant-size output invites the reading that the model has compressed the text into 6 KB, and that reading breaks in both directions. It is not lossless: the runbook cannot be reconstructed from its 1,536 floats by any decoder the provider ships, and the vector carries nothing like the information a gzip of the same document does. It is also not as lossy as people assume — research on embedding inversion has shown that short texts can be recovered from their vectors with high fidelity by an iterative model trained for the purpose, which matters if you were treating a vector index as a safe place to put data you would not put in plaintext.

This is where the hash analogy from the opening has to be paid for, because it is right about exactly one thing and wrong about the rest. A hash and an embedding agree that output size is fixed while input size is not, and that is the whole of the resemblance. A hash is designed so that two nearly identical inputs produce completely unrelated outputs; an embedding is designed for the precise opposite, and the 0.887 above is that design working. A hash is also designed to be irreversible, and the inversion research says an embedding is not reliably so. Treat the analogy as a statement about the shape of the function signature and nothing further.

Nor is a vector an identifier. Teams occasionally reach for one as a deduplication key on the grounds that identical text produces identical floats, and it does, but only within a single model build on a single hardware and precision configuration; the same text embedded through a GPU batch of size 64 and again on its own can differ in the last few decimal places, which is enough to break exact-match lookup while leaving cosine at 0.99999. Equality is not the operation these things support, and the deduplication you want is a threshold over a similarity search, which is a different query with different failure modes.

The fixed output length also does not mean the input length is free. Every model has a maximum sequence length, and text beyond it is truncated rather than rejected. all-MiniLM-L6-v2 cuts at 256 word pieces in its trained configuration, bge-base-en-v1.5 at 512, text-embedding-3-small at 8,191. Feed a 3,000-token chunk to a 512-token model and you get a perfectly valid 384-dimensional vector describing the first sixth of the chunk, with no error and no warning, and every query that should have matched the second half of the document silently misses. That fraction is the truncation limit divided by the chunk length, and it is the number the fixed output width conceals: the vector is the right shape whatever share of the text reached the encoder. The row in the table above where the runbook truncated to 512 tokens is that failure, written down.

Dilution is the subtler cost. As a chunk grows, its pooled vector drifts toward whatever the average of its content is, and the average of four subjects is a point that answers none of the four well. Splitting one 800-word runbook section covering restart, rollback, log retrieval and escalation into four chunks of roughly 200 words each raised the cosine between a rollback query and the chunk that answers it from 0.48 to 0.71 in the corpus this chapter has been using. The vector length was 1,536 in both cases, which is exactly why chunking is a question about content and not about storage. Note which direction that runs: the split raised the score, and it did so by removing the other three subjects from the average rather than by adding anything about rollback.

So the length is settled, and the two laws that govern a corpus are separate. The count of chunks times a constant gives the bytes; the count of tokens times a price gives the bill; and a chunking decision moves both of them in directions that have to be reasoned about one at a time. Anyone sizing a vector index from the length of the text has fused the two together, and the resulting figure is wrong by whatever factor the corpus happens to supply.

What has not been explained is why the paraphrase landed at 0.887 in the first place. The pooling story says how many tokens contributed and with what weight; it says nothing about why those particular coordinates ended up where they did. Answering that is what tells you which other pairs of sentences will land close and which will not — including several that ought to be opposites.