Attention and the Transformer: The Architecture Behind Modern AI

This is the sixth post in our AI & ML Foundations series, building on embeddings. Start with What Is Machine Learning, Really? if you're joining partway through.

Raj built the internal tool that summarized customer support tickets for a logistics company, so a human agent could scan a two-line summary instead of reading a full back-and-forth thread. It worked, mostly — until a ticket like this one came through:

"The courier left the package outside in the rain even though the customer specifically asked him not to. He apologized, but she wants a refund."

Raj's early summarizer, built on an older architecture that processed words strictly one at a time, in order, sometimes generated: "The customer apologized for leaving the package in the rain." Backwards. It had lost track of which "he" belonged to which earlier noun by the time it reached the end of the sentence — a genuinely common failure for models that process language sequentially, because by word twenty, whatever the model "remembered" about word two has often faded into noise.

This is the exact problem the Transformer architecture — the "T" in GPT, and the foundation underneath nearly every serious language model in production today — was built to solve directly.

The problem with reading strictly left to right

Before Transformers, most language models processed a sentence the way you'd read it with your finger under each word — one at a time, carrying forward a single running summary of everything seen so far. That running summary is a genuine bottleneck: by the time the model reaches "he apologized," it needs to still be holding onto the fact that "he" refers to "the courier," mentioned twelve words earlier, while several other nouns ("package," "customer") have come and gone in between. A single, fixed-size running memory just isn't built for that.

The fix: let every word look directly at every other word

The Transformer's core mechanism, self-attention, throws out the sequential bottleneck entirely. Instead of relying on a fading running memory, every single word in a sentence gets to look directly at every other word, all at once, and decide — for itself — which of them actually matter for understanding it.

Crucially, nobody tells the model in advance that "he" should attend to "courier" — that weighting is learned during training, the same gradient-descent process covered earlier in this series, by seeing enough real sentences that the correct pattern of attention consistently produces lower loss than the wrong one.

The mechanism, concretely: query, key, value

Self-attention computes, for every word, three vectors derived from its embedding: a query ("what am I looking for?"), a key ("what do I represent, that something else might be looking for?"), and a value ("what information do I actually contribute, if attended to?").

import numpy as np
 
def self_attention(query, keys, values):
    # how well does this word's query match every other word's key?
    scores = np.dot(keys, query)
    weights = np.exp(scores) / np.sum(np.exp(scores))  # softmax, turning scores into a distribution
    return np.dot(weights, values)  # weighted blend of every word's value
 
# simplified: "he" query, against keys for "courier", "package", "customer"
he_query = np.array([0.9, 0.1])
keys = np.array([[0.85, 0.15], [0.2, 0.7], [0.3, 0.6]])   # courier, package, customer
values = np.array([[1.0, 0.0], [0.0, 1.0], [0.0, 0.5]])
 
self_attention(he_query, keys, values)
# array([0.71, 0.29]) — the result leans heavily toward "courier"'s value

That softmax step is the same operation that turned raw scores into a genuine probability distribution for the classifiers earlier in this series — here, it turns "how well does my query match each key" into a set of attention weights that sum to 1, which is exactly the mixture used to compute the final blended representation. The word "he" doesn't end up as a fixed vector anymore; it ends up as a contextual blend, mostly "courier," with a little of everything else nearby.

Multi-head attention: looking for several kinds of relationship at once

A single attention mechanism captures one kind of relationship well — pronoun resolution, say. Real language has many kinds of relationship happening simultaneously in the same sentence: subject-verb agreement, cause and effect, sentiment ("angry" attaching to "she," not "he"). Multi-head attention runs several independent attention mechanisms in parallel, each free to specialize in a different kind of relationship, then combines their results:

from tensorflow.keras.layers import MultiHeadAttention
 
attention_layer = MultiHeadAttention(num_heads=8, key_dim=64)
output = attention_layer(query=sequence, key=sequence, value=sequence)

Eight independent "attention heads" here, each learning its own pattern of what to focus on — one might specialize in resolving pronouns, another in tracking sentiment, another in verb-argument structure — with no human assigning those specializations directly; they emerge from training, the same way the meaning-space structure in the previous post's embeddings emerged from training rather than manual design.

Why this actually fixed Raj's summarizer

Once Raj's summarization tool moved to a Transformer-based model, the pronoun-confusion failure essentially disappeared — not because anyone patched a special case for pronouns, but because every word in the sentence now has direct, unmediated access to every other word, with no fading running-memory bottleneck standing between "he" and "courier," however far apart they sit in the actual sentence.

Input: "The courier left the package outside in the rain even though the
       customer specifically asked him not to. He apologized, but she
       wants a refund."

Output: "Courier failed to follow delivery instructions; customer
        requesting refund."

Positional encoding: attention alone doesn't know word order

There's a genuine gap self-attention leaves open, worth naming directly: comparing every word against every other word, all at once, means the mechanism described so far has no inherent sense of order — "the courier left the package" and "the package left the courier" would look identical to attention alone, since it's just comparing the same set of words against each other regardless of position. Positional encoding fixes this by adding information about each word's position directly into its embedding before attention ever runs:

import numpy as np
 
def positional_encoding(position, d_model):
    angles = position / np.power(10000, (2 * np.arange(d_model // 2)) / d_model)
    return np.concatenate([np.sin(angles), np.cos(angles)])
 
pos_0 = positional_encoding(0, 512)  # encodes "this is the 1st word"
pos_5 = positional_encoding(5, 512)  # encodes "this is the 6th word"

Adding this positional signal directly to each word's embedding is what lets the model tell "courier left package" apart from "package left courier" — without it, the powerful all-at-once comparison in Figure 1 would have thrown away word order entirely, a genuinely different sentence meaning lost along with it.

Why this same mechanism scales so well

The practical reason Transformers displaced sequential architectures isn't just accuracy — it's that self-attention's "every word compares to every other word at once" computation parallelizes cleanly across modern GPU hardware, since every one of those comparisons is independent and can run simultaneously. A sequential model, by contrast, genuinely has to finish processing word one before it can start word two, because each step depends on the previous one's output. That difference is a large part of why Transformer-based models could be trained on vastly more text than earlier architectures in a comparable amount of wall-clock time — the architecture itself, not just more data or bigger hardware, unlocked training at the scale that produces today's large language models.

What to actually remember from this post

• Sequential models process words one at a time through a single fading running memory — which is exactly why they lose track of long-distance references like pronouns.
• Self-attention lets every word look directly at every other word simultaneously, learning which ones actually matter for understanding it — no sequential bottleneck at all.
• Query, key, and value vectors, combined via softmax-weighted blending, are the actual mechanism — and the attention weights are learned from data, never hand-specified.
• Multi-head attention runs several attention mechanisms in parallel, each free to specialize in a different kind of relationship within the same sentence.
• This mechanism is the actual architectural foundation of GPT and nearly every modern language model — not an incremental tweak to older approaches, but a genuinely different way of processing sequences.

Next in the series: Tokenization: How Text Becomes Numbers, where we go back to the very first step — before attention, before embeddings — and look at how a sentence gets split into pieces in the first place.