Mid · Core path · 6 chapters
What an embedding is
Dense vector, semantic space, dimensionality
Embedding intuition is foundational for retrieval, but treating vector proximity as truth causes silent misses, false matches, and brittle indexes.
Read these first
What you will know
Embed a sentence and inspect the vector
An embedding is a fixed-length vector positioning text in a space where nearby means similar.
Show two paraphrases landing close together
The result depends on the model and similarity metric, so a convincing example should include the measured distance or rank.
Compare coordinate values for related texts and test whether one coordinate tracks a labeled property
Coordinates are usually distributed features rather than independently meaningful fields, so apparent patterns may not generalize.
Compute similarity for a claim, its negation, and an unrelated sentence, then inspect the ranking
Dense spaces often place opposing statements near each other because shared topic dominates the missing logical relation.
Measure nearest-neighbor overlap before and after reducing the vectors to fewer dimensions
Projection can discard distinctions that matter to retrieval even when a plot still looks visually coherent.
Chapters
- 1
The same vector length comes back for five words or five hundred
Embed a sentence, print the array, and confirm two things at once: its length is a property of the model rather than of the input, because per-token representations are pooled into a single vector; and a paraphrase of that sentence lands measurably close to it. Report the cosine and the rank rather than the impression, since "these are close" means nothing without a number and a named model.
- 2
The encoder was trained to pull paraphrases together and never shown a negation
"Nearby means similar" is not a fact about geometry, it is a summary of what the training objective rewarded: pairs that co-occurred or paraphrased one another were pulled together, randomly sampled unrelated pairs were pushed apart. Nothing in that procedure ever presented a sentence beside its own negation as a pair to separate, so the space carries every axis for what two sentences are about and none for the logical relation between them. Knowing where the geometry came from is what turns the chapters that follow from observations to memorise into predictions you can make, because it tells you in advance which distinctions the space preserves and which it was never asked to.
- 3
Coordinates are distributed features, not named fields
Coordinate 42 running unusually large across every pricing document looks like a discovery and is almost always noise, because models spread a feature across many coordinates instead of assigning it a slot. The test that separates the two cases is to correlate that coordinate against a labelled property on a held-out sample and watch the apparent pattern fail to generalise.
- 4
Shared topic drowns negation, which is why 0.94 is correct
Cosine 0.94 between "the API returns 200 on success" and its exact negation is the model behaving as designed: the two sentences share nearly all their content, and the objective never trained the relation that separates them. Neither escape route works either — on normalised vectors squared Euclidean distance is a monotone function of cosine, so changing the metric reproduces the same ranking exactly, and extra dimensions buy capacity rather than a relation the training never contained. Take that to a corpus of operational runbooks full of do and do-not instructions, where every remedy sits outside the vector: an entailment check or reranker over the retrieved set, chunking so an instruction and its negation never share a chunk, or a lexical constraint on negation terms carried alongside the dense query.
- 5
A clean 2D plot has already thrown the distances away
Squeezing 1536 dimensions into two forces the projection to distort distance, so crisply separated clusters by document type establish nothing about whether the right passage gets retrieved. Measure nearest-neighbour overlap before and after any reduction instead. You will also learn to spot the one feature of such a plot that genuinely should worry you when a PM wants it in a review deck.
- 6
A cosine score is calibrated to one model and one index
Vectors from two different models are coordinates in unrelated spaces, so the cosine between them is a number with no interpretation at all. That makes a hard-coded 0.75 cutoff a property of one model over one corpus, and re-embedding only new documents after a model swap leaves a single index holding two incompatible geometries. Order the resulting harms by how quickly each one starts hurting.
External resources
- LLMs are Also Effective Embedding Models: An Inarxiv.org
An in-context learning approach for using LLMs as embedding models, with comparative retrieval evaluations.
- From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Languagearxiv.org
Dense vector representations across word, contextual and multimodal models, from semantic spaces to applications.
- Embeddings: Dense Vector Representations of Discrete Dataml4devs.com
This is a substantive explainer that covers dense vectors, similarity, training, static versus contextual embeddings, pretrained models, and practical dimensionality considerations, despite some oversimplifications and dated model claims.
- Embeddings in Machine Learning: An Overviewlightly.ai
Why nearby vectors mean similar meaning, and what dimensionality buys you
- Understanding the role of embeddings in RAG LLMsaporia.com
Explains dense vectors, semantic space, and dimensionality in retrieval-augmented generation.
- Mastering RAG: How to Select an Embedding Modelrungalileo.io
Explains dense vectors, semantic spaces, and dimensionality for selecting embedding models in RAG.
- What are Embeddings in Machine Learning?cloudflare.com
Explains embeddings as dense vectors, semantic spaces, and dimensionality in machine learning.
- What is Embedding?ibm.com
Explains embeddings as dense vectors in semantic spaces and introduces dimensionality.
- An Introduction to Embedding Vectors | Christoph HenkelmannMLCon | Community & Conferences
Engineer-facing intro to what a dense vector actually encodes
- What Are Word Embeddings?Under The Hood
IBM short on semantic space, giving the mental model in ten minutes
- What are Word Embeddings?IBM Technology
Explains how words become dense vectors in semantic space and why dimensionality matters.