Tokenization: How Text Becomes Numbers

This is the seventh post in our AI & ML Foundations series, building on attention and the Transformer. Start with What Is Machine Learning, Really? if you're joining partway through.

Sofia was building a writing assistant on top of a large language model, and the bug report that landed on her desk was strange enough that she assumed it was a joke at first: "how many letter Rs are in the word 'strawberry'?" — a question the assistant confidently, incorrectly answered. Not once. Consistently. The same model that could write a technically sound essay about strawberry cultivation couldn't reliably count the letters in the word itself.

Once she dug in, the explanation turned out to be simpler and stranger than any actual reasoning failure: the model had never seen the individual letters "s-t-r-a-w-b-e-r-r-y" at all. Before any of the attention mechanics from the previous post ever run, text has to be split into chunks called tokens — and "strawberry" isn't ten letter-tokens. It's likely two or three larger chunks, and the model genuinely cannot count something it was never shown as separate pieces in the first place.

Why not just use individual characters?

The most obvious approach — one token per character — would actually solve Sofia's specific bug. It creates a much bigger problem in exchange: a Transformer's attention mechanism, from the previous post, compares every token against every other token, and that comparison cost grows sharply with sequence length. A paragraph that's 200 words as word-tokens might be 1,000+ individual characters — a five-times-longer sequence for the exact same amount of actual content, which is both slower to process and harder for the model to find long-range patterns across.

Why not just use whole words either?

The opposite extreme — one token per whole word — has its own real problem: language keeps producing new words. A model trained with a fixed vocabulary of whole words has no way to represent "cryptocurrency" if that word didn't exist yet when the vocabulary was built, and no graceful way to handle typos, made-up words, or words from other languages mixed into a sentence.

The actual answer: pieces smaller than words, bigger than letters

Modern tokenizers split text into subword pieces — chunks larger than individual letters but often smaller than whole words, chosen so that common words stay as one token while rare or unfamiliar words get broken into a small number of familiar pieces:

import tiktoken
 
encoding = tiktoken.encoding_for_model("gpt-4o")
 
tokens = encoding.encode("strawberry")
print(tokens)
# [496, 675, 19772]
 
for t in tokens:
    print(t, "->", encoding.decode([t]))
# 496 -> "straw"
# 675 -> "ber"
# 19772 -> "ry"

"Strawberry" becomes three tokens — straw, ber, ry — none of which is a single letter, and none of which lines up neatly with "count the Rs." The model processes these three chunks, not ten characters, and the "R" that appears twice inside the "ber" and "ry" tokens is buried inside pieces the model was never asked to count characters within. It's not that the model can't count. It's that it's fundamentally working with a different, coarser unit than the one the question is actually about.

How the tokenizer actually decides where to split

The specific algorithm behind most modern tokenizers is Byte-Pair Encoding (BPE): start with individual characters as the smallest unit, then repeatedly merge whichever adjacent pair of units appears most frequently across a massive body of real text, building up progressively larger common chunks over many merge rounds. "The" gets merged into a single token almost immediately, because it's everywhere. Genuinely rare technical terms end up as several smaller pieces, because there was never enough repeated evidence to justify merging them into one.

common_word = encoding.encode("the")
print(common_word)          # [1820] — a single token, extremely common
rare_word = encoding.encode("floccinaucinihilipilification")
print(len(rare_word))       # 12 tokens — genuinely rare, split into many small pieces

This is the direct, practical reason a rare or technical word costs measurably more, in the token-based pricing covered elsewhere on this blog, than a common one of similar length — it isn't charged per letter or per "word" in the everyday sense, but per token, and rare words simply need more of them.

The consequence Sofia actually had to design around

Once she understood the real cause, Sofia's fix wasn't trying to make the model "better at counting" through a bigger model or a cleverer prompt — that fundamentally can't work around a representation that doesn't expose individual letters at all. Instead, for any user question that genuinely required character-level reasoning, she routed it to actual code:

def count_letter(word: str, letter: str) -> int:
    return word.lower().count(letter.lower())
 
count_letter("strawberry", "r")  # 3

Handing character-counting to a simple, deterministic function — rather than asking a token-based language model to do something its own representation makes structurally difficult — is the same "use the right tool for the actual problem" instinct behind function calling, covered elsewhere on this blog: a model reasoning in tokens is genuinely the wrong tool for a task that's fundamentally about characters, no matter how large or capable the model gets.

Why this matters well beyond one quirky bug

Tokenization quietly explains several other things worth knowing:

• Why token counts, not word counts, determine cost and context-window limits — the token-per-price billing covered in our OpenAI series is counting these exact subword chunks, not English words.
• Why the same text can tokenize differently across languages. Languages with less representation in a tokenizer's training data often split into noticeably more tokens per word than English does, for text carrying the same amount of actual meaning — a real, measurable cost and context-budget difference between languages.
• Why a leading or trailing space can change a word's token entirely — " strawberry" and "strawberry" are frequently different tokens or token sequences, not the same token with a space glued on, a genuinely common source of subtle prompt-engineering bugs.

Special tokens: the vocabulary entries that aren't words at all

Beyond ordinary subword pieces, a tokenizer's vocabulary reserves specific entries for structural purposes a model needs to track — where a sequence begins and ends, where one speaker's turn stops and another's starts:

tokens = encoding.encode("<|im_start|>user\nHow many Rs in strawberry?<|im_end|>")

These special tokens (exact names vary by model family) aren't part of natural language at all — they're markers the model was specifically trained to recognize as structural boundaries, the same way HTML tags mark structure in a document without being part of the visible text. This is worth knowing because it explains why a raw prompt string sent to a chat-based model is quietly wrapped in this kind of scaffolding before it ever reaches the model — the plain text you write is rarely the literal token sequence the model actually processes.

Checking token counts before you're surprised by a bill or a truncated response

Because tokens — not words or characters — are the actual unit of both cost and context-window limits (covered in depth in our OpenAI series), it's worth checking a piece of text's real token count directly, rather than estimating from word count:

def count_tokens(text: str, model: str = "gpt-4o") -> int:
    encoding = tiktoken.encoding_for_model(model)
    return len(encoding.encode(text))
 
count_tokens("strawberry")           # 3
count_tokens("the quick brown fox")  # 4 — common words, roughly one token each

Notice "strawberry" alone costs as many tokens as the entire four-word phrase "the quick brown fox" — a genuinely useful thing to know before assuming word count is a reasonable proxy for either cost or how much of a model's context window a piece of text will actually consume.

What to actually remember from this post

• Text is split into subword tokens before a model ever sees it — a middle ground between one-token-per-character (too slow, too long) and one-token-per-word (can't handle new or rare words).
• Byte-Pair Encoding builds a vocabulary by repeatedly merging the most frequent adjacent pairs in a large body of text — common words become single tokens, rare words fragment into several.
• A model reasoning over tokens genuinely cannot see individual letters directly — character-level tasks (counting letters, precise spelling manipulation) are a poor fit for token-based reasoning, and are better handled by actual code.
• Token count, not word or character count, is the real unit behind cost and context-window limits — and it varies by language and by how common a given word actually is.

Next in the series: Training vs. Inference: What Actually Happens When You Ask an LLM a Question, where we separate the (slow, expensive, occasional) process of building a model from the (fast, cheap, constant) process of actually using one.