Overfitting, Underfitting, and the Bias-Variance Tradeoff

This is the fourth post in our AI & ML Foundations series. Start with What Is Machine Learning, Really? if you're joining partway through.

Naomi ran demand forecasting for a regional grocery chain — predicting how many units of each product each store would need to order, so shelves stayed stocked without drowning a backroom in unsold inventory. Her first real model, trained on eighteen months of historical sales, was genuinely spectacular against that same historical data: 99.2% accuracy, errors so small her manager assumed she'd made a mistake in the evaluation code.

Then it went live, and it was terrible. A three-day heatwave sent ice cream sales up 40%, and the model barely reacted. A competitor closed nearby and foot traffic redistributed overnight — the model kept forecasting as if nothing had changed. It wasn't that the model had learned "how demand responds to weather and competition." It had learned the eighteen specific months in its training data, essentially by heart, and anything genuinely new looked like static to it.

This is overfitting, and understanding it precisely — not just as "the model got it wrong," but as a specific, predictable failure mode — is arguably the single most practically important idea in this entire series.

Two ways to fail, in opposite directions

There's a matching failure on the other side, worth naming just as clearly: underfitting, where a model is too simple to capture even the patterns genuinely present in the data.

The middle model in Figure 1 is the actual goal — it captures the real underlying trend and deliberately ignores the noise around it. The one on the right achieves a lower error on this specific dataset than the one in the middle, and is dramatically worse in practice, because it wired the random noise into its predictions as if it were signal.

Seeing it directly in Naomi's own numbers

The tell, once you know to look for it, is a gap between two numbers that should otherwise track each other:

from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeRegressor
 
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
 
model = DecisionTreeRegressor(max_depth=None)  # unrestricted depth — free to memorize
model.fit(X_train, y_train)
 
print("Training accuracy:", model.score(X_train, y_train))
print("Test accuracy:", model.score(X_test, y_test))
# Training accuracy: 0.998
# Test accuracy: 0.71

A 0.998 training score paired with a 0.71 test score is the single clearest overfitting signature there is. This is precisely why the first post in this series insisted on holding back test data before training even started — without that held-out set, Naomi's original model would have looked flawless right up until the exact moment it started making decisions that cost the business real money.

Bias and variance: naming the two failure directions precisely

Bias is error from a model being too simple to capture the real pattern — the underfitting case, systematically wrong in a consistent direction, no matter what data it's shown. Variance is error from a model being too sensitive to the specific data it happened to train on — the overfitting case, where a slightly different training set would have produced a meaningfully different model.

Both failure modes produce a model that performs poorly on new data — but they call for opposite fixes. A high-bias model needs more capacity (a bigger network, more informative features). A high-variance model needs less capacity, or some other constraint on how freely it's allowed to fit — which is exactly what Naomi reached for next.

Regularization: deliberately handicapping the model

Rather than letting a model fit the training data as tightly as it possibly can, regularization adds a penalty for complexity, forcing the model to only use that complexity where it genuinely earns its keep:

from sklearn.linear_model import Ridge
 
model = Ridge(alpha=1.0)  # alpha controls how strongly complexity is penalized
model.fit(X_train, y_train)

alpha trades training-set fit for genuine robustness — a larger value forces smaller, more conservative weights throughout the model, which tends to produce a worse training score and a better test score, precisely the direction Naomi actually needed to move in. For neural networks specifically, dropout achieves something similar by a different mechanism: during training, it randomly disables a fraction of neurons on each pass, which prevents the network from becoming overly reliant on any single neuron memorizing one specific quirk of the training set.

from tensorflow.keras import layers
 
model = keras.Sequential([
    layers.Dense(128, activation="relu"),
    layers.Dropout(0.3),  # randomly zero out 30% of neurons during each training step
    layers.Dense(64, activation="relu"),
    layers.Dense(1),
])

Early stopping: catching the exact moment it turns from learning to memorizing

The clearest real-time signal of overfitting shows up by plotting training loss against validation loss (measured on held-out data, never used for weight updates) across epochs:

history = model.fit(
    X_train, y_train,
    validation_data=(X_val, y_val),
    epochs=100,
)

Both losses fall together at first — the model is genuinely learning the real pattern. At some point, training loss keeps falling while validation loss starts climbing back up: the model has stopped learning generalizable structure and started memorizing training-set-specific noise. Early stopping simply halts training the moment validation loss stops improving, rather than running the full epoch count regardless:

from tensorflow.keras.callbacks import EarlyStopping
 
early_stop = EarlyStopping(monitor="val_loss", patience=5, restore_best_weights=True)
model.fit(X_train, y_train, validation_data=(X_val, y_val), epochs=100, callbacks=[early_stop])

This is the single most direct fix Naomi actually applied — her original model had simply trained for far longer than it should have, well past the point where it stopped learning genuine seasonal patterns and started memorizing the specific noise in eighteen particular months.

What to actually remember from this post

• Overfitting means memorizing training-specific noise instead of learning the real pattern — recognizable by a large gap between training performance and held-out test performance.
• Underfitting means the model is too simple to capture even the real pattern that exists — both training and test performance are poor, together.
• Bias and variance pull in opposite directions, and total error is lowest at a genuine middle ground — more capacity fixes bias; more constraint fixes variance.
• Regularization (like Ridge's alpha, or dropout in neural networks) deliberately constrains a model so it can't fit the training data as freely as it otherwise would.
• Early stopping catches the exact point where training loss and validation loss diverge — the real-time signature of a model crossing from learning into memorizing.

Next in the series: Embeddings: Teaching Machines What Words Mean, where the "features" problem from the first post in this series gets genuinely hard — because words don't have obvious numbers hiding inside them the way pixels do.