Deep Learning Explained: Neural Networks from Zero to Understanding

Deep Learning Explained: Neural Networks from Zero to Understanding

A neural network is a parameterized function: it takes an input, passes it through layers of matrix multiplications and nonlinearities, and produces an output. Most tutorials skip straight to model.fit() and never explain why any of it works — so the moment your loss plateaus, you have no idea what to try.

This guide starts with the math instead, not to torture you but because the math is what makes debugging possible. Once you know what a network is actually computing, every training failure becomes diagnosable rather than mysterious.

The Real Definition of a Neural Network

Training a neural network means searching for the parameters, or weights, that make its output match what you want — nothing more mystical than that.

The mechanism is gradient descent: measure how wrong the network is with a loss function, compute how much each weight contributed to that wrongness (the gradient), and nudge each weight slightly in the direction that reduces the error. Repeat thousands of times and the wrongness shrinks.

This idea predates deep learning by decades. Rosenblatt's perceptron (1957) did this for a single layer. What changed in the 1980s was the discovery — or rediscovery — of how to efficiently compute gradients through multiple stacked layers. That algorithm is backpropagation, and it's still exactly what PyTorch runs under the hood today.

Forward Pass: What the Network Computes

Take a single neuron. It computes a weighted sum of its inputs, adds a bias term, and passes the result through an activation function:

output = activation(w₁x₁ + w₂x₂ + ... + wₙxₙ + b)

In matrix notation, for an entire layer with m neurons receiving n inputs:

z = Wx + b        # linear transformation, shape [m]
a = activation(z) # elementwise nonlinearity, shape [m]

Stack several of these layers and you have a "deep" network — each layer takes the previous layer's output and reshapes it into something more useful for the final prediction, the way each stage of an assembly line refines the product a little further.

Here is a minimal NumPy implementation to make this concrete:

import numpy as np

def relu(z):
    return np.maximum(0, z)

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

class DenseLayer:
    def __init__(self, in_features, out_features):
        # Xavier initialization — prevents vanishing/exploding gradients
        scale = np.sqrt(2.0 / in_features)
        self.W = np.random.randn(out_features, in_features) * scale
        self.b = np.zeros(out_features)
    
    def forward(self, x):
        self.x = x          # cache for backward pass
        self.z = self.W @ x + self.b
        self.a = relu(self.z)
        return self.a

Xavier initialization exists to solve a specific failure mode: weights that start too large make activations explode as you go deeper, and weights too small make them vanish to zero. Scaling weights by sqrt(2/fan_in) keeps the variance of activations roughly constant across layers — a result derived by Glorot and Bengio in their 2010 paper.

Why Depth Matters: The Hierarchy of Representations

"Deep" in deep learning refers to depth of composition — how many layers stack on top of each other — not depth in any mystical sense. Each layer learns a different level of abstraction.

In an image-recognition network, the first layers detect edges and colors, middle layers detect textures and parts (an eye, a wheel), and final layers detect whole objects and scenes. This hierarchy emerges automatically from training; nobody tells layer 3 to look for eyes.

That claim isn't speculation. Zeiler and Fergus (2014) visualized exactly what each layer of a CNN responds to by finding the images that maximize specific neurons' activations, and the edges-to-parts-to-objects hierarchy showed up directly in the pictures.

The advantage of depth is computational, not just conceptual. A shallow, two-layer network computing a certain function might need exponentially many neurons to do it. A deeper network can compute the same function with far fewer total parameters by reusing intermediate computations across layers — the same reason circuit complexity favors depth over width.

Loss Functions: Defining "Wrong"

A loss function is the single number that quantifies how wrong a network's prediction is — you cannot fix a mistake you haven't measured.

For binary classification:

Binary Cross-Entropy = -(y log(ŷ) + (1-y) log(1-ŷ))

For multi-class classification (softmax output):

Categorical Cross-Entropy = -Σᵢ yᵢ log(ŷᵢ)

For regression:

Mean Squared Error = (1/n) Σ (y - ŷ)²

The choice of loss function encodes what you actually care about. Mean squared error (MSE) penalizes large errors quadratically — a prediction off by 10 hurts 100x more than one off by 1, like a fine that scales with the square of how late you are. For tasks where outliers shouldn't dominate, Huber loss blends L1 and L2 smoothly, capping the quadratic penalty.

Backpropagation: The Chain Rule Applied

Backpropagation is the algorithm that computes ∂L/∂W for every weight matrix W in the network by applying the calculus chain rule backward through the layers — the part most tutorials skip or hand-wave.

Gradient descent then updates each weight: W ← W - lr * ∂L/∂W.

The chain rule says: if L = f(g(x)), then dL/dx = (dL/df) * (df/dg) * (dg/dx).

For a two-layer network with layers z₁ = W₁x, a₁ = relu(z₁), z₂ = W₂a₁, L = loss(z₂):

∂L/∂W₂ = ∂L/∂z₂ · ∂z₂/∂W₂ = δ₂ · a₁ᵀ

∂L/∂W₁ = ∂L/∂a₁ · ∂a₁/∂z₁ · ∂z₁/∂W₁
        = (W₂ᵀδ₂) · relu'(z₁) · xᵀ

Where relu'(z) = 1 if z > 0 else 0 — the gradient of ReLU.

The critical insight is that gradients flow backward through the exact same path the forward computation used, reusing cached activations along the way. That's why self.x = x appears in the forward pass above — backprop needs it later.

Here is the backward pass for the dense layer:

class DenseLayer:
    # ... (forward pass as above)
    
    def backward(self, dL_da):
        # Gradient of ReLU: zero where pre-activation was negative
        dL_dz = dL_da * (self.z > 0).astype(float)
        
        # Gradient w.r.t. weights and bias
        self.dL_dW = np.outer(dL_dz, self.x)   # shape [out, in]
        self.dL_db = dL_dz                       # shape [out]
        
        # Gradient to pass to previous layer
        dL_dx = self.W.T @ dL_dz                # shape [in]
        return dL_dx

Notice how dL_dx — the gradient passed backward — depends on the transpose of the weight matrix. Geometrically, the forward pass projects through W, and the backward pass projects back through Wᵀ. The symmetry is elegant.

Network Architecture: Visualizing the Flow

This is a standard fully-connected (dense) network for MNIST digit classification. Each arrow represents a weight matrix. The network has roughly 784×256 + 256×128 + 128×64 + 64×10 ≈ 250,000 parameters.

Training Loop in PyTorch

PyTorch handles the gradient computation automatically via autograd. Understanding what it's doing under the hood (as we just covered) lets you debug it when things go wrong.

import torch
import torch.nn as nn
from torch.utils.data import DataLoader
from torchvision import datasets, transforms

# --- Model Definition ---
class MLP(nn.Module):
    def __init__(self):
        super().__init__()
        self.network = nn.Sequential(
            nn.Flatten(),
            nn.Linear(784, 256),
            nn.ReLU(),
            nn.Dropout(0.3),           # regularization
            nn.Linear(256, 128),
            nn.ReLU(),
            nn.Dropout(0.3),
            nn.Linear(128, 64),
            nn.ReLU(),
            nn.Linear(64, 10)
            # No softmax here — CrossEntropyLoss includes it
        )
    
    def forward(self, x):
        return self.network(x)

# --- Data ---
transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.1307,), (0.3081,))  # MNIST mean/std
])
train_data = datasets.MNIST('./data', train=True, download=True, transform=transform)
train_loader = DataLoader(train_data, batch_size=64, shuffle=True)

# --- Training ---
model = MLP()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.CrossEntropyLoss()

def train_epoch(model, loader, optimizer, criterion):
    model.train()
    total_loss, correct = 0, 0
    
    for batch_x, batch_y in loader:
        # Forward pass
        logits = model(batch_x)
        loss = criterion(logits, batch_y)
        
        # Backward pass — PyTorch computes all gradients
        optimizer.zero_grad()   # clear gradients from last step
        loss.backward()         # backprop
        
        # Gradient clipping prevents exploding gradients
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
        
        optimizer.step()        # gradient descent step
        
        total_loss += loss.item()
        correct += (logits.argmax(dim=1) == batch_y).sum().item()
    
    n = len(loader.dataset)
    return total_loss / len(loader), correct / n

for epoch in range(10):
    loss, acc = train_epoch(model, train_loader, optimizer, criterion)
    print(f"Epoch {epoch+1:2d} | Loss: {loss:.4f} | Acc: {acc:.4f}")

The training loop diagram makes the flow explicit:

Activation Functions: Why Not Just Linear?

An activation function is the nonlinearity applied after each linear layer — and without it, stacking layers is pointless, because the product of matrices is just another matrix. No matter how many layers you stack, a purely linear network collapses to one matrix multiplication; depth adds zero expressive power.

The "dead ReLU" problem is worth understanding on its own: when a ReLU neuron's input is always negative, its output is always zero, its gradient is always zero, and it never learns again — permanently stuck, like a light switch wired backward. A learning rate that's too high can push many neurons into this state at once. Leaky ReLU and GELU avoid it by never fully zeroing the gradient.

Regularization: Preventing Memorization

Regularization is any technique that stops a network from simply memorizing the training set — a network with enough parameters can hit 100% training accuracy while learning nothing generalizable, the equivalent of a student who memorizes exam answers instead of the underlying concept.

• Dropout randomly zeroes out neurons during training (Srivastava et al., 2014): this stops neurons from co-adapting, since no neuron can count on specific other neurons always being present. At inference, outputs are scaled by the keep probability.
• Weight decay (L2 regularization) penalizes large weights: adding λ||W||² to the loss encourages the network to spread evidence across many features rather than leaning on a few.
• Batch normalization stabilizes training itself (Ioffe & Szegedy, 2015): normalizing activations within each mini-batch to zero mean and unit variance, then applying a learnable scale and shift, smooths the loss landscape enough to allow higher learning rates and less sensitivity to initialization.

# BatchNorm in practice — usually placed after Linear, before activation
nn.Sequential(
    nn.Linear(256, 128),
    nn.BatchNorm1d(128),   # normalize across batch dimension
    nn.ReLU(),
)

Common Benchmark Results

To ground your expectations, here are results on standard benchmarks for fully-connected networks vs. the deeper architectures we'll cover in subsequent articles:

The jump from logistic regression to a 4-layer MLP — 5.7 percentage points — comes purely from learning hierarchical nonlinear features. The further jump to CNNs comes from incorporating the spatial structure of images, which we cover in the next article.

Practical Debugging Checklist

When your network is not learning, work through this list:

1. Loss is not decreasing at all: check learning rate (try 10x higher/lower), check that optimizer.zero_grad() is called, verify labels are correct
2. Loss decreases then plateaus: try learning rate scheduling (torch.optim.lr_scheduler.CosineAnnealingLR), add more capacity, check for data quality issues
3. Training loss low but validation loss high: add dropout, weight decay, data augmentation, or collect more training data
4. Loss is nan: learning rate too high, check for log(0) in your loss, check for inf in inputs
5. Accuracy stuck at 1/num_classes: class imbalance, or the optimizer is not receiving gradients (check requires_grad)

These diagnostics come from understanding the math. Without it, you're guessing.

Where to Go From Here

Fully-connected networks are the foundation, but they are rarely the right architecture for specific data types. Images have spatial structure that MLPs ignore — convolutional networks exploit it explicitly. Sequences have temporal dependencies that MLPs cannot handle — recurrent networks and transformers capture them.

Check out the Deep Learning Quiz to test your understanding of these concepts, then continue to Convolutional Neural Networks Explained to see how architecture shapes learning.

For the mathematical foundations, the ML Algorithms Quiz covers the key concepts, and the Machine Learning course provides structured practice.

The ML category has additional articles on specific architectures and techniques, and if you want to understand how transformers fit in, transformer architecture notes are a good companion to this material.