Embeddings: Teaching Machines What Words Mean

This is the fifth post in our AI & ML Foundations series. Start with What Is Machine Learning, Really? if you're joining partway through.

Elena owned search quality at a mid-size online shoe retailer, and the support ticket that finally got her a real budget to fix things was almost embarrassing in its simplicity: a customer had searched "cheap running shoes," gotten zero results, and left a one-star review — even though the store had exactly the product, listed under "affordable trainers." The search engine was matching on exact keywords, character by character, and "cheap" and "affordable" don't share a single letter in the same position. As far as the system was concerned, they were as unrelated as "cheap" and "kayak."

Every technique from earlier in this series — feature vectors, neural networks, gradient descent — assumed you already had sensible numbers to feed a model. Pixels are naturally numeric: brightness, straightforwardly, is already a number. Words aren't. "Affordable" and "cheap" look nothing alike as strings of characters, despite meaning almost the same thing, while "affordable" and "afford" look extremely similar as strings and are grammatically related but not interchangeable in a product search. Whatever numeric representation Elena needed had to somehow capture meaning, not spelling — and that representation is called an embedding.

The core idea: distance should mean similarity

An embedding represents a word (or a sentence, or an entire document) as a list of numbers — a vector — positioned in space such that words with similar meaning end up close together, and words with unrelated meaning end up far apart. Not close in spelling. Close in meaning.

from sentence_transformers import SentenceTransformer
 
model = SentenceTransformer("all-MiniLM-L6-v2")
 
cheap_vec = model.encode("cheap running shoes")
affordable_vec = model.encode("affordable trainers")
kayak_vec = model.encode("bright red kayak")
 
print(len(cheap_vec))  # 384

384 numbers, for a five-word phrase — none of them individually meaningful to a person reading them, the same way no single pixel of a photo tells you what's in the picture. What matters is the vector's position, relative to other vectors.

Measuring "how similar" with actual math

Cosine similarity measures the angle between two vectors — a value from -1 (opposite) to 1 (identical direction), with values near 1 meaning "these point in essentially the same direction in meaning-space":

import numpy as np
 
def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
 
cosine_similarity(cheap_vec, affordable_vec)   # 0.87 — genuinely close
cosine_similarity(cheap_vec, kayak_vec)         # 0.11 — essentially unrelated

Notice what actually happened here: cheap_vec and affordable_vec share almost no characters at all, and they landed at 0.87 — properly close. cheap_vec and kayak_vec share letters and even a similar length, and landed at 0.11 — properly far apart. This is the entire fix for Elena's ticket: search stops comparing character strings and starts comparing meaning positions.

Where these numbers actually come from

Nobody hand-designs the 384 dimensions in Elena's embedding model. They emerge from training a neural network (using exactly the gradient descent and backpropagation mechanics covered earlier in this series) on a genuinely enormous amount of text, with a training objective built around one core intuition: words that show up in similar contexts tend to mean similar things. "Cheap" and "affordable" both frequently appear near "budget," "discount," and "price" in real writing; "kayak" almost never does. That statistical regularity, extracted automatically across billions of sentences, is what ultimately produces vectors positioned the way Figure 1 shows.

Arithmetic that actually works on meaning

The famous demonstration that convinced a lot of skeptical engineers embeddings were capturing something real:

king = model.encode("king")
man = model.encode("man")
woman = model.encode("woman")
 
result_vector = king - man + woman
 
# result_vector lands extremely close to model.encode("queen")

Subtracting "man" from "king" and adding "woman" produces a vector landing almost exactly where "queen" independently sits. Nobody programmed a rule about royal titles and gender — the model discovered, purely from patterns in how these words are actually used in text, that "king is to man as queen is to woman" is a consistent, almost linear relationship in the space it built. That's the moment embeddings stop feeling like a clever numerical trick and start feeling like the model actually captured something structurally true about meaning.

Using embeddings to fix Elena's actual search problem

product_catalog = ["affordable trainers", "premium leather boots", "waterproof hiking boots", "budget running shoes"]
catalog_vectors = model.encode(product_catalog)
 
query_vector = model.encode("cheap running shoes")
similarities = [cosine_similarity(query_vector, v) for v in catalog_vectors]
 
ranked = sorted(zip(similarities, product_catalog), reverse=True)
for score, product in ranked:
    print(f"{score:.2f}  {product}")

0.89  budget running shoes
0.84  affordable trainers
0.31  waterproof hiking boots
0.22  premium leather boots

Elena's original keyword search would have returned nothing for this query, or ranked by accident. The embedding-based version correctly surfaces the two genuinely relevant products first — not because either one shares words with the query, but because their meaning does. This exact mechanism, ranking by embedding similarity rather than keyword overlap, is also the retrieval step underneath every RAG (retrieval-augmented generation) system covered elsewhere on this blog: the same technique that fixed Elena's search box is what lets a chatbot find the right paragraph in a company's documentation, even when a user's question shares no words with it at all.

Embeddings for more than single words

Everything in this post has used short phrases as examples, but the same technique scales to entire sentences, paragraphs, or documents — a sentence embedding compresses an entire piece of text into one vector, positioned in the same kind of meaning-space as the word-level example above:

review_1 = model.encode("These shoes fell apart after two weeks of light use.")
review_2 = model.encode("The stitching came undone within a month of normal wear.")
review_3 = model.encode("Fast shipping and great customer service overall.")
 
cosine_similarity(review_1, review_2)  # 0.81 — both are about durability failing
cosine_similarity(review_1, review_3)  # 0.14 — unrelated complaint

Two reviews sharing almost no words at all — "fell apart," "stitching came undone" — land close together because they're describing the same underlying problem. This is exactly how a real product-feedback system groups thousands of free-text reviews into themes automatically, without anyone writing keyword rules for every possible way a customer might phrase "this broke too soon."

A genuine limitation worth knowing: embeddings reflect their training data

An embedding model's sense of "similar" is entirely a product of the text it was trained on — which means it can encode real, sometimes unwanted associations right along with genuinely useful ones. A model trained predominantly on English retail listings, for instance, may position regional product terms or non-English brand names much further from their true synonyms than an equally well-understood English term would land. This isn't a bug to be "fixed" so much as a property to actively check for: evaluating an embedding model against examples that matter for your specific domain and user base — not just the generic "king/queen" demo — is worth doing before trusting it in a system that makes real decisions, the same evaluation discipline covered in depth later in this series.

What to actually remember from this post

• An embedding is a numeric vector positioned so that distance reflects meaning, not spelling — the fix for exactly the "cheap vs. affordable" problem keyword matching can't solve.
• Cosine similarity measures how close two vectors are in direction — a practical, computable stand-in for "how similar are these two things in meaning."
• These vectors emerge from training on real text, using the same gradient-descent mechanics from earlier in this series — nobody hand-designs what each dimension means.
• Vector arithmetic on embeddings often captures genuine relationships (king − man + woman ≈ queen) — a sign the model learned real structure in meaning, not just a lookup table.
• This is the same mechanism behind modern semantic search and RAG retrieval — ranking by meaning-similarity rather than exact keyword overlap.

Next in the series: Attention and the Transformer, where embeddings for individual words aren't quite enough — because what a word means often depends on which other words are sitting next to it.