Convolutional Neural Networks (CNNs): How Image Recognition Works

Convolutional Neural Networks (CNNs): How Image Recognition Works

A convolutional neural network (CNN) is a neural network that hard-codes two assumptions about images directly into its architecture: nearby pixels are related, and the same visual pattern means the same thing no matter where it sits in the frame. Most explanations stop at "CNNs use sliding filters" and move on — that skips the actual insight.

Think of a CNN as a detective who only needs to learn what a fingerprint looks like once, then can spot that same fingerprint anywhere on the page. A regular network would have to relearn it separately for every possible location.

Those two priors — locality and translation invariance — are why CNNs need far fewer parameters than a generic network and why they became the default architecture for vision.

Why Fully-Connected Networks Fail for Images

A fully-connected layer treats every pixel as unrelated to every other pixel, which throws away the one fact you know for certain about images: neighboring pixels belong together.

A 224×224 RGB image has 224 × 224 × 3 = 150,528 input values. A single fully-connected hidden layer with 4,096 neurons would need 150,528 × 4,096 ≈ 617 million parameters — just for the first layer. AlexNet (Krizhevsky et al., 2012), the network that reset the field of computer vision, has only 60 million total parameters, and most of that budget went elsewhere.

Beyond the parameter count, fully-connected layers ignore spatial structure entirely. Every input pixel connects to every neuron with an independent weight. The network has no built-in way to notice that adjacent pixels form edges, or that an eye looks the same in the top-left of an image as in the bottom-right — it has to learn each case from scratch, separately, at a staggering parameter cost.

Convolutional layers solve both problems at once.

The Convolution Operation

A convolution slides a small weight matrix — the kernel, or filter — across the input and computes a dot product at every position, the way a stamp presses the same design onto every page of a notebook. If the input is a 2D feature map I and the kernel is K of size k×k:

(I * K)[i, j] = Σₘ Σₙ I[i+m, j+n] · K[m, n]

The output is called a feature map or activation map. Each value answers one question: how strongly does this kernel's pattern appear at this location?

The key mechanism is weight sharing: the same kernel weights K are applied at every position. A filter that detects a horizontal edge uses the same nine numbers, for a 3×3 kernel, whether it looks at position (0,0) or position (100,100). This reduces parameters from H × W × k × k, the fully-connected cost, down to just k × k.

A single convolutional layer applies multiple filters, each learning to detect a different pattern — one for edges, one for curves, one for a particular texture. With 64 filters of size 3×3×3 applied to an RGB image:

Parameters = 64 × (3 × 3 × 3) + 64 (bias) = 1,792

Compare that to the millions a fully-connected layer would need.

Convolution in PyTorch

import torch
import torch.nn as nn
import torch.nn.functional as F

# A single convolutional layer
conv = nn.Conv2d(
    in_channels=3,    # RGB input
    out_channels=64,  # 64 different filters
    kernel_size=3,    # 3x3 kernels
    padding=1,        # 'same' padding — output same spatial size as input
    stride=1
)

# Manual convolution to see the math
x = torch.randn(1, 3, 32, 32)  # batch_size=1, channels=3, 32x32
output = conv(x)
print(f"Input shape:  {x.shape}")        # [1, 3, 32, 32]
print(f"Output shape: {output.shape}")   # [1, 64, 32, 32]

# Visualizing what a learned filter looks like
# After training, conv.weight[0] is the first filter
# Shape: [3, 3, 3] — 3 input channels, 3x3 spatial

Pooling: Downsampling with Purpose

Pooling is a layer that shrinks a feature map's spatial size by summarizing each local neighborhood into a single value — the CNN equivalent of zooming out to see the bigger picture instead of every brick in the wall. Max pooling takes the maximum value in each local window:

pool = nn.MaxPool2d(kernel_size=2, stride=2)
# Halves spatial dimensions: [1, 64, 32, 32] → [1, 64, 16, 16]

Max pooling wins over the alternatives for two reasons: it tolerates small shifts — if a feature moves slightly within the pooling window, the maximum still fires — and it cuts computation, since every later layer processes fewer spatial positions.

Global average pooling (GAP) averages across the entire spatial dimension, collapsing a feature map to a single value per channel. GAP replaced fully-connected layers in modern architectures, cutting millions of parameters to zero while improving generalization — proof that fewer parameters can mean a better model, not a weaker one.

Building a CNN from Scratch

class ConvNet(nn.Module):
    """
    A small CNN for CIFAR-10 (32x32 RGB, 10 classes).
    Architecture: 3 conv blocks + global average pooling + classifier
    """
    def __init__(self):
        super().__init__()
        
        # Block 1: 3 → 64 channels, 32x32 → 16x16
        self.block1 = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True),
            nn.Conv2d(64, 64, kernel_size=3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(2, 2)   # 32x32 → 16x16
        )
        
        # Block 2: 64 → 128 channels, 16x16 → 8x8
        self.block2 = nn.Sequential(
            nn.Conv2d(64, 128, kernel_size=3, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(inplace=True),
            nn.Conv2d(128, 128, kernel_size=3, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(2, 2)   # 16x16 → 8x8
        )
        
        # Block 3: 128 → 256 channels, 8x8 → 4x4
        self.block3 = nn.Sequential(
            nn.Conv2d(128, 256, kernel_size=3, padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(inplace=True),
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(2, 2)   # 8x8 → 4x4
        )
        
        # Global average pooling: 4x4 → 1x1
        self.gap = nn.AdaptiveAvgPool2d(1)
        
        # Classifier
        self.classifier = nn.Sequential(
            nn.Flatten(),
            nn.Dropout(0.5),
            nn.Linear(256, 10)
        )
    
    def forward(self, x):
        x = self.block1(x)
        x = self.block2(x)
        x = self.block3(x)
        x = self.gap(x)
        return self.classifier(x)

model = ConvNet()
print(f"Parameters: {sum(p.numel() for p in model.parameters()):,}")
# ~1.1M parameters — lean but effective

Architecture Evolution

Each architecture introduced an idea that became standard practice:

• AlexNet popularized ReLU: it swapped tanh for ReLU activations, added dropout regularization, and proved GPU training could scale.
• VGGNet chose simplicity: deep stacks of plain 3×3 convolutions beat hand-tuned exotic kernel shapes.
• GoogLeNet added multi-scale vision: Inception modules process the input at several receptive-field sizes simultaneously.
• ResNet solved depth: skip connections are the single most important CNN innovation, discussed next.
• EfficientNet scaled systematically: compound scaling grows width, depth, and resolution together instead of tuning one at a time.

Residual Networks: Solving the Depth Problem

A residual network (ResNet) is a CNN that adds a shortcut path around each block, letting the block learn only the correction to its input rather than the whole transformation from scratch.

Before ResNets, deeper networks were paradoxically worse than shallower ones — not from overfitting, but from optimization difficulty. Gradients vanished as they backpropagated through many layers. A 56-layer network trained worse than a 20-layer network on the training set itself (He et al., 2016), which ruled out overfitting as the culprit.

The residual block solves this elegantly:

class ResidualBlock(nn.Module):
    """
    Basic ResNet residual block (He et al., 2016).
    F(x) + x: the block learns the residual, not the full transformation.
    """
    def __init__(self, channels):
        super().__init__()
        self.conv1 = nn.Conv2d(channels, channels, 3, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(channels)
        self.conv2 = nn.Conv2d(channels, channels, 3, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(channels)
    
    def forward(self, x):
        identity = x                           # skip connection
        
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))        # no activation yet
        
        out = out + identity                   # add skip connection
        out = F.relu(out)                      # activate after addition
        
        return out

The skip connection does two things. At initialization, F(x) is near zero, so F(x) + x ≈ x — the block starts as an identity mapping and only has to learn small deviations from it, which is far easier than learning a full transformation from scratch. During backprop, the gradient flows through two paths at once: through the residual branch F(x) and directly through the skip connection. Even if the residual branch's gradient vanishes, the skip connection keeps information flowing — like a highway bypass that stays open even when the local streets are jammed.

ImageNet Benchmark Results

ImageNet is the standard benchmark for image classification: 1.2 million training images across 1,000 classes. Top-1 accuracy is the fraction of images where the model's top prediction is correct.

EfficientNet-B4 beats ResNet-152 on accuracy with a third of the parameters — proof that architecture design matters as much as raw scale. The FLOP count matters for deployment specifically: a model that needs 37 GFLOPs cannot run in real time on a mobile device, no matter how accurate it is.

Receptive Field: How Much Does Each Neuron See?

The receptive field is the region of the original input that influences a single neuron's output — and in a deep network it is much larger than that neuron's own kernel size suggests.

Think of it like standing progressively farther back from a mosaic: each step back lets your eye take in more tiles at once, even though your vision hasn't physically changed.

For a stack of n layers with kernel size k and no pooling:

Receptive field = 1 + n × (k - 1)

With pooling (stride 2) every p layers:

Effective receptive field grows as ~2^(n/p)

This is why networks go deep: each layer compounds the previous one, letting final layers integrate information from the entire image while early layers stay focused on fine-grained local patterns.

After five layers of 3×3 convolutions with 2×2 max pooling every two layers, a neuron can "see" an area of roughly 64×64 pixels — far beyond the 3×3 patch its own weights cover.

Data Augmentation: Making the Most of Limited Data

Data augmentation is the practice of generating modified copies of your training images — flipped, cropped, recolored — so a limited dataset behaves like a larger, more diverse one. CNNs are hungry for data, and augmentation is the cheapest way to feed that hunger:

from torchvision import transforms

train_transform = transforms.Compose([
    transforms.RandomCrop(32, padding=4),         # random crop with padding
    transforms.RandomHorizontalFlip(p=0.5),        # horizontal flip
    transforms.ColorJitter(
        brightness=0.2, contrast=0.2,
        saturation=0.2, hue=0.1
    ),
    transforms.RandomRotation(degrees=15),
    transforms.ToTensor(),
    transforms.Normalize(
        mean=[0.485, 0.456, 0.406],               # ImageNet statistics
        std=[0.229, 0.224, 0.225]
    ),
])

# Validation — no augmentation, only normalization
val_transform = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

Each transform exploits a known invariance: a dog is still a dog when flipped horizontally, so horizontal flip is safe; lighting shouldn't change object identity, so color jitter teaches that; scale and position shouldn't matter either, so random crops teach that too.

Using a Pre-Trained ResNet

Transfer learning means starting from a backbone already trained on ImageNet instead of training from scratch — almost always the right default, since training ImageNet-scale models from zero requires hundreds of GPU-hours and millions of images.

import torchvision.models as models

# Load pre-trained ResNet-50
backbone = models.resnet50(weights=models.ResNet50_Weights.IMAGENET1K_V2)

# Freeze early layers — they have learned general features
for name, param in backbone.named_parameters():
    if 'layer4' not in name and 'fc' not in name:
        param.requires_grad = False

# Replace the classification head for your number of classes
num_classes = 5  # your task
backbone.fc = nn.Sequential(
    nn.Dropout(0.3),
    nn.Linear(backbone.fc.in_features, num_classes)
)

# Only the final layers are trainable
trainable = sum(p.numel() for p in backbone.parameters() if p.requires_grad)
total = sum(p.numel() for p in backbone.parameters())
print(f"Trainable: {trainable:,} / {total:,} ({100*trainable/total:.1f}%)")

Freezing the early layers keeps the features that transfer well and only spends training budget on the layers specific to your task. For more detail on fine-tuning strategies, see the Transfer Learning Explained article.

Practical Training Tips

A few habits separate models that train cleanly from ones that stall:

• Learning rate warmup avoids early destructive updates: start with a small learning rate — about a tenth of the target — for the first few epochs, then ramp up linearly, since weights are still randomly initialized and large steps can wreck them.
• Cosine annealing finds flatter, more generalizable minima: decay the learning rate along a cosine curve instead of a step schedule.

scheduler = torch.optim.lr_scheduler.OneCycleLR(
    optimizer,
    max_lr=0.01,
    epochs=50,
    steps_per_epoch=len(train_loader),
    pct_start=0.3,    # 30% of training for warmup
)

• Mixed precision training cuts memory and speeds up training: run the forward and backward pass in float16 and keep parameter updates in float32, which nearly halves memory use and often speeds up training 2-3x on modern GPUs.

scaler = torch.cuda.amp.GradScaler()

with torch.cuda.amp.autocast():
    output = model(input)
    loss = criterion(output, target)

scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()

From CNNs to Transformers

CNNs dominated computer vision from AlexNet in 2012 through roughly 2020. Then Vision Transformers (ViT) (Dosovitskiy et al., 2021) showed that transformers could match or beat CNNs on ImageNet — but only when trained on enough data.

The competition is unresolved: ConvNeXt (Liu et al., 2022) showed that modernizing CNNs with transformer-inspired design choices closes most of the accuracy gap. Both architectures remain relevant, and hybrid designs combining both are increasingly common in production systems.

For sequence problems — text, audio, time series — CNNs play a secondary role. That territory belongs to recurrent networks and transformers, covered in LSTM vs Transformer.

Test your CNN knowledge with the Deep Learning Quiz, and for more context on how these architectures fit together, the ML Algorithms Quiz covers the broader landscape.

The Machine Learning course includes hands-on CNN projects, and LLM concepts notes explain how visual features connect to language model representations.