Sigmoid:    σ(x) = 1 / (1 + e^(-x))
Tanh:       tanh(x) = (e^x - e^(-x)) / (e^x + e^(-x))
ReLU:       f(x) = max(0, x)
Leaky ReLU: f(x) = max(αx, x)  where α = 0.01
ELU:        f(x) = x if x > 0, else α(e^x - 1)
GELU:       f(x) = x · Φ(x)   where Φ is CDF of normal distribution
SiLU/Swish: f(x) = x · σ(x)
Softmax:    f(xᵢ) = e^xᵢ / Σe^xⱼ

import torch
import torch.nn as nn

# Apply in layers
layer = nn.Sequential(
    nn.Linear(128, 64),
    nn.ReLU(),              # or nn.GELU(), nn.SiLU()
    nn.Linear(64, 10),
    nn.Softmax(dim=-1)      # for multiclass output
)

# Functional API
import torch.nn.functional as F
x = F.relu(x)
x = F.gelu(x)
x = F.softmax(logits, dim=-1)

ReLU(x) = 0 for all x ≤ 0
→ gradient = 0 → weights stop learning

import torch.nn as nn

# Mean Squared Error
criterion_mse = nn.MSELoss()

# Mean Absolute Error
criterion_mae = nn.L1Loss()

# Huber (SmoothL1)
criterion_huber = nn.SmoothL1Loss(beta=1.0)

loss = criterion_mse(predictions, targets)
loss.backward()

# Binary classification (sigmoid output)
criterion = nn.BCELoss()
# or combined (more numerically stable):
criterion = nn.BCEWithLogitsLoss()   # applies sigmoid internally

# Multiclass (softmax output)
criterion = nn.CrossEntropyLoss()    # applies log_softmax internally
# Usage: CrossEntropyLoss expects raw logits (not softmax) + integer labels
loss = criterion(logits, targets)    # targets: torch.long integers

# Class-weighted loss for imbalanced data
weights = torch.tensor([1.0, 3.0, 2.0])  # weight minority classes higher
criterion = nn.CrossEntropyLoss(weight=weights)

Regression?
  ├─ Normal noise → MSE
  ├─ Outliers present → MAE or Huber
  └─ Distribution prediction → NLL / Gaussian NLL

Classification?
  ├─ Binary output → BCEWithLogitsLoss
  ├─ Multiclass (mutually exclusive) → CrossEntropyLoss
  ├─ Multilabel (multiple can be true) → BCEWithLogitsLoss per label
  └─ Extreme class imbalance → Focal Loss

Embeddings / Similarity?
  ├─ Two classes (similar/different) → Contrastive Loss
  ├─ Ranked similarity → Triplet Loss
  └─ Dense retrieval → InfoNCE / NT-Xent

class FocalLoss(nn.Module):
    def __init__(self, gamma=2.0, alpha=0.25):
        super().__init__()
        self.gamma = gamma
        self.alpha = alpha

    def forward(self, logits, targets):
        ce = F.binary_cross_entropy_with_logits(logits, targets, reduction='none')
        p_t = torch.exp(-ce)
        loss = self.alpha * (1 - p_t) ** self.gamma * ce
        return loss.mean()