Transformer Architecture Cheat Sheet

What Is a Transformer?

Transformers are the neural network architecture behind every modern LLM GPT-4, Claude, Gemini, LLaMA. Introduced in the 2017 paper Attention Is All You Need, they replaced recurrent networks (RNNs/LSTMs) with a fully attention based approach that parallelizes training and handles long range dependencies.

Core Components

Self Attention Formula

Attention(Q, K, V) = softmax( QKᵀ / √dₖ ) · V

• Q (Query) what this token is looking for
• K (Key) what each token offers
• V (Value) what each token contributes if selected
• √dₖ scale factor to prevent large dot products from vanishing gradients in softmax

Multi head attention runs h attention heads in parallel, then concatenates:

MultiHead(Q,K,V) = Concat(head₁,...,headₙ) · Wᴼ
head_i = Attention(QWᵢᵠ, KWᵢᴷ, VWᵢᵛ)

Encoder vs Decoder

Encoder Decoder (T5, BART): Encoder reads input, decoder generates output ideal for translation, summarization.

Attention Mask Types

Key Hyperparameters

Positional Encoding

Sinusoidal (original paper):

PE(pos, 2i)   = sin(pos / 10000^(2i/d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

Rotary Position Embedding (RoPE) used in LLaMA, Mistral:

• Encodes position as rotation in embedding space
• Better extrapolation to longer sequences than sinusoidal

ALiBi used in MPT, BLOOM:

• Adds a linear bias to attention scores based on distance

Transformer Block (One Layer)

x → LayerNorm → Multi-Head Attention → + x (residual)
  → LayerNorm → Feed-Forward Network  → + x (residual)

Pre LN (normalize before sublayer) is more stable than Post LN for training very deep models.

Scaling Laws (Chinchilla)

Loss scales predictably with compute: L ∝ (N/Nₒ)^(-αN) + (D/Dₒ)^(-αD) where N = parameters, D = data.

Efficient Attention Variants

Mini PyTorch Attention

import torch
import torch.nn.functional as F

def scaled_dot_product_attention(q, k, v, mask=None):
    d_k = q.size(-1)
    scores = torch.matmul(q, k.transpose(-2, -1)) / d_k ** 0.5
    if mask is not None:
        scores = scores.masked_fill(mask == 0, -1e9)
    weights = F.softmax(scores, dim=-1)
    return torch.matmul(weights, v)

Common Mistakes

• Forgetting to scale by √dₖ softmax gradients vanish with large dot products
• Confusing encoder attention (bidirectional) with decoder attention (causal/masked)
• Using Post LN in very deep models Pre LN trains more stably
• Not masking padding tokens they pollute attention scores
• Conflating model size (parameters) with capability data quality and training recipe matter equally