Generative Models: From GPT to Diffusion

This is the tenth and final post in our AI & ML Foundations series, building on everything from what machine learning is through neural networks, training, overfitting, embeddings, attention, tokenization, training vs. inference, and evaluation.

Leo was scoping a new feature for a marketing creative tool — a panel where a user could type a product description and get both marketing copy and a matching product image generated automatically. Two AI-generated outputs, side by side, from what he'd assumed was basically the same underlying technology applied to two content types. His engineering lead corrected him gently: "the text model and the image model don't just use different data. They're solving the generation problem in fundamentally different ways."

That distinction — not "same idea, different data" but "genuinely different mechanisms" — is worth understanding clearly, because it explains a lot of what feels mysterious about generative AI once you actually see both approaches side by side.

Text generation: one token at a time, never revising

Every language model covered throughout this series — built from the Transformer architecture, tokenizing input into subword pieces, using attention to weigh context — generates text the exact same way: predict the single most probable next token, append it, then repeat, using everything generated so far (including its own previous output) as new context for the next prediction.

def generate_autoregressively(prompt, model, max_tokens=50):
    tokens = tokenize(prompt)
    for _ in range(max_tokens):
        next_token_probs = model.predict_next_token(tokens)
        next_token = sample_from(next_token_probs)
        tokens.append(next_token)
    return detokenize(tokens)

This is called autoregressive generation — each new token depends on everything before it, generated strictly in order, left to right, with no mechanism to go back and revise an earlier word once it's been committed. Every single "next word prediction" you've watched stream in from a chatbot is exactly this loop, running once per token.

Image generation: starting from noise, refining the whole picture at once

A diffusion model — the kind of architecture underneath most modern AI image generators — solves the problem the opposite way. Instead of building an image piece by piece in a fixed order, it starts with an image of pure random noise and repeatedly refines the entire image, all at once, over many steps, gradually removing noise until a coherent picture emerges.

def generate_image_via_diffusion(prompt, model, num_steps=50):
    image = random_noise(size=(512, 512, 3))
    for step in reversed(range(num_steps)):
        predicted_noise = model.predict_noise(image, prompt, step)
        image = remove_noise_estimate(image, predicted_noise, step)
    return image

Every one of the 50 steps touches the whole image simultaneously — there's no "left pixel decided before right pixel" ordering at all. The model was trained to do exactly one thing well: given a noisy image and a description of what it should eventually become, predict what noise was added, so it can be subtracted out. Repeat that fifty times, and pure static gradually resolves into a coherent picture.

Why this difference isn't just academic

The practical consequences Leo actually needed to plan around, once his lead walked him through it:

• Text generation can't easily "go back" mid-response. Because each token depends on everything generated before it, a language model that starts a sentence in a way that becomes awkward has no built-in mechanism to revise earlier words — it can only keep generating forward, which is part of why "regenerate the whole response" is the practical fix, not "edit just this part," when output goes wrong.
• Diffusion naturally supports partial editing. Because every step touches the whole image, tools built on diffusion models can mask out just one region (a customer's product photo background, say) and re-run the noise-removal process on just that region while leaving the rest untouched — a genuinely different editing capability than autoregressive text generation offers.
• The two have different quality/speed trade-offs. More diffusion steps generally means a higher-quality, more refined image, at a roughly proportional cost in time — a real, tunable dial. Autoregressive text generation's length is mostly determined by how much text is actually needed, not a similar quality/steps trade-off in the same way.

Where the two approaches actually meet

Modern multimodal systems increasingly combine both: a Transformer-based model (autoregressive, token-by-token) handles the language understanding — reading Leo's product description, extracting what should actually appear in the image — and hands a structured prompt off to a diffusion model to handle the actual pixel generation. Two fundamentally different generative mechanisms, from everything covered in this post, working together in the same product feature, each doing the part it's genuinely suited for.

What this series covered, start to finish

Ten posts, and a genuine arc: machine learning's core reversal (rules as output, not input) led into neural networks (stacking simple units to represent curves), how they actually learn (gradient descent and backpropagation), and the specific way that learning can go wrong (overfitting and underfitting). From there, embeddings solved the "how do you turn meaning into numbers" problem, attention solved "how do you track relationships across a whole sentence," and tokenization covered the actual units a model reasons over. Training vs. inference separated building a model from using one, and evaluation covered how to actually tell whether it's any good. This post closes the loop by showing how those same underlying ideas — layers, training, learned representations — get assembled two genuinely different ways, depending on whether the output is a sentence or a picture.

What to actually remember from this post

• Autoregressive text generation predicts one token at a time, using everything generated so far as context, strictly left to right, with no revision of earlier tokens.
• Diffusion image generation starts from pure noise and refines the entire image simultaneously, over many steps, rather than building it in a fixed order.
• This mechanical difference has real practical consequences — diffusion naturally supports masked, partial editing; autoregressive text generation generally has to regenerate from scratch to fix an earlier mistake.
• Modern multimodal tools often combine both — an autoregressive model handling language understanding, handing off to a diffusion model for the actual image synthesis — each used for what it's genuinely suited to.

If your team is building on top of these same fundamentals — whichever architecture actually fits the problem — that's exactly the kind of hands-on, first-principles work we build our AI & Generative AI training around.