Neural Networks: Teaching Silicon to Recognize Patterns

This is the second post in our AI & ML Foundations series. Start with What Is Machine Learning, Really? if you're joining partway through.

Dan ran the camera-trap network for a wildlife conservancy — forty motion-triggered cameras scattered across a forest reserve, each one dumping thousands of nighttime photos onto an SD card every month. Someone had to look at every single image and tag what was in it: deer, fox, raccoon, or nothing but wind-blown grass setting off the motion sensor. That someone was, increasingly, Dan, at 11 p.m., squinting at grainy infrared photos.

He tried the obvious first fix: simple rules based on brightness patterns and blob size — "if the bright shape is taller than it is wide and appears low in the frame, it's probably a fox." It worked reasonably well right up until a raccoon standing on its hind legs produced almost exactly that same shape, and a photo of nothing but a swaying branch occasionally tripped every rule at once. A fox and a raccoon, in silhouette, at 2 a.m., in grainy infrared, simply don't separate along any straight-line rule Dan could invent. The boundary between them wasn't a line. It needed to bend.

Why one straight line stops being enough

The kind of model from the first post in this series — one weighted sum, compared against a threshold — can only ever draw a straight line (or a flat plane, in higher dimensions) through feature space, no matter how its weights are tuned. Some problems are separable that way. Telling a fox from a raccoon in a dim, grainy photo isn't.

The perceptron: one artificial neuron

The building block that eventually fixes this is deliberately simple. A perceptron takes several numeric inputs, multiplies each by a weight, adds them up along with a bias term, and passes the result through an activation function:

import numpy as np
 
def perceptron(inputs, weights, bias):
    weighted_sum = np.dot(inputs, weights) + bias
    return 1 if weighted_sum > 0 else 0  # a step activation function
 
inputs = np.array([0.8, 0.3, 0.9])   # e.g., pixel-brightness features from a crop of the photo
weights = np.array([0.5, -0.2, 0.7])
bias = -0.4
 
perceptron(inputs, weights, bias)  # 1

On its own, this is just the same weighted-sum idea from the first post in this series with extra steps — one straight-line boundary. The actual idea only shows up once you stop using one perceptron and start stacking them.

Stacking perceptrons: where curves come from

A neural network is layers of these units, each layer's output feeding the next layer's input. A hidden layer sits between the raw input and the final prediction — and it's what actually gives the network the ability to represent curves and genuinely complex shapes, not just straight lines.

Why the activation function is the part that actually matters

Here's the detail that trips people up the first time: if every neuron just computed a weighted sum with no activation function, stacking layers would be pointless — mathematically, a stack of purely linear operations collapses back into one single linear operation, no matter how many layers you add. The non-linear activation function applied after each neuron's weighted sum is what actually gives depth its power. A modern network almost always uses ReLU:

def relu(x):
    return np.maximum(0, x)

relu just zeroes out anything negative and passes anything positive through unchanged — a strange-looking choice that turns out to work remarkably well in practice, and is dramatically cheaper to compute than the smooth curves earlier networks relied on.

Building the actual network

Here's the classifier Dan actually built, using Keras — the point isn't to memorize this API, it's to see the layer-stacking idea from Figure 2 turn into a handful of lines of real code:

from tensorflow import keras
from tensorflow.keras import layers
 
model = keras.Sequential([
    layers.Dense(256, activation="relu", input_shape=(4096,)),  # hidden layer
    layers.Dense(64, activation="relu"),                          # a second hidden layer
    layers.Dense(4, activation="softmax"),                        # output: deer, fox, raccoon, empty
])
 
model.compile(optimizer="adam", loss="sparse_categorical_crossentropy", metrics=["accuracy"])
model.fit(train_photos, train_labels, epochs=8, validation_split=0.15)

4096 is a small, downsampled 64×64 grayscale crop of the motion-triggered region of each photo, flattened into one long vector — the same "turn the real thing into numbers" step from the first post in this series, just with far more numbers this time. Four outputs, one per category, each a probability — softmax is what guarantees those numbers actually add up to 1, so "the model's confidence across categories" is a genuine probability distribution, not just independent scores.

Reading the output correctly

predictions = model.predict(new_photo)
print(predictions)
# [0.03, 0.09, 0.85, 0.03]   -> deer, fox, raccoon, empty
 
np.argmax(predictions)  # 2 (raccoon)

The network doesn't output "it's a raccoon." It outputs "here's how confident I am in each category," and argmax — simply picking the highest number — is what turns that into a single answer. This distinction matters in practice: a prediction of [0.03, 0.09, 0.85, 0.03] (fairly confident) and one of [0.24, 0.26, 0.28, 0.22] (genuinely unsure, everything nearly tied) can both argmax to "raccoon," while representing completely different levels of trust a real system should place in that answer — and for Dan, a low-confidence prediction was exactly the signal worth flagging for a human to double-check, rather than trusting blindly.

What "deep" actually refers to

"Deep learning" is simply this same idea — layers of neurons, non-linear activations between them — with more layers, often dozens or hundreds in a real production system, each learning to recognize progressively more abstract patterns: early layers might respond to edges and simple textures, middle layers to shapes like ears or tails, later layers to something closer to "this silhouette pattern looks like a fox." Nobody designs those intermediate representations by hand — they emerge from training, which is exactly what the next post in this series covers: the actual mechanism by which a network with millions of essentially-random starting weights gradually turns into one that reliably tells a fox from a raccoon in the dark.

What to actually remember from this post

• A single perceptron is just a weighted sum with a threshold — one straight-line boundary. Nothing new happens until you stack them.
• Hidden layers plus non-linear activation functions are what let a network represent curves, not just straight lines — stacking purely linear layers would collapse back into one linear operation.
• A network's output is a probability distribution, not a single answer — argmax picks the winner, but the full distribution tells you how confident that pick actually was, and low confidence is itself useful information.
• "Deep" just means "many layers" — the same basic building block from this post, repeated, learning progressively more abstract patterns at each layer.

Next in the series: How Neural Networks Learn: Gradient Descent and Backpropagation, where a network's millions of starting weights actually turn into something that works.