LLM Context Window Explained: Why It Matters and How to Use It

LLM Context Window Explained: Why It Matters and How to Use It

When GPT-3 launched in 2020 with a 4,096-token context window, it felt limiting — you couldn't even process a single long article without chunking. Today, GPT-4 Turbo offers 128K tokens. Gemini 1.5 Pro extends to 1 million tokens, enough to process an entire novel in one prompt.

The context window is one of the most practically important properties of any LLM — it determines what you can do in a single conversation, how long documents you can analyze, and how much code the model can consider at once. But bigger isn't always better, and knowing when to use long context versus chunking or RAG is an important practical skill.

Context Windows Across Major Models (2025)

Token Counting in Practice

import tiktoken

# GPT-4 tokenizer
enc = tiktoken.encoding_for_model("gpt-4")

# Count tokens in different content types
examples = {
    "Short text": "The quick brown fox jumps over the lazy dog.",
    "Technical code": """
def fibonacci(n):
    if n <= 1:
        return n
    a, b = 0, 1
    for _ in range(2, n + 1):
        a, b = b, a + b
    return b
    """,
    "Long paragraph": """
    Machine learning is a subset of artificial intelligence that provides 
    systems the ability to automatically learn and improve from experience 
    without being explicitly programmed. Machine learning focuses on the 
    development of computer programs that can access data and use it to 
    learn for themselves.
    """
}

for name, text in examples.items():
    tokens = enc.encode(text)
    words = len(text.split())
    ratio = len(tokens) / max(words, 1)
    print(f"{name}: {words} words → {len(tokens)} tokens (ratio: {ratio:.2f})")

# Estimate context budget
def estimate_context_usage(system_prompt, conversation_history, documents):
    total = 0
    total += len(enc.encode(system_prompt))
    for message in conversation_history:
        total += len(enc.encode(message))
    for doc in documents:
        total += len(enc.encode(doc))
    return total

print(f"\nRemaining context for generation: {128000 - total} tokens")

Managing Context Limits

Strategy 1: Sliding Window for Conversations

When conversation history exceeds the context window:

from typing import List
import tiktoken

class ContextManager:
    def __init__(self, model="gpt-4o", max_tokens=100000):
        self.enc = tiktoken.encoding_for_model(model)
        self.max_tokens = max_tokens
        self.messages = []
        self.system_prompt = ""
    
    def count_tokens(self, messages: List[dict]) -> int:
        total = len(self.enc.encode(self.system_prompt))
        for msg in messages:
            total += len(self.enc.encode(msg.get("content", ""))) + 4  # message overhead
        return total
    
    def add_message(self, role: str, content: str):
        self.messages.append({"role": role, "content": content})
        
        # If over limit, prune from the middle (keep first turn + recent turns)
        while self.count_tokens(self.messages) > self.max_tokens:
            # Keep first user message for context, remove second-oldest
            if len(self.messages) > 2:
                self.messages.pop(1)  # Remove second message, preserve first
            else:
                break
    
    def get_messages(self) -> List[dict]:
        return [{"role": "system", "content": self.system_prompt}] + self.messages

ctx = ContextManager()
ctx.system_prompt = "You are a helpful assistant."
ctx.add_message("user", "What is machine learning?")
ctx.add_message("assistant", "Machine learning is...")

Strategy 2: Hierarchical Summarization

For very long documents, summarize before including:

from openai import OpenAI

client = OpenAI()

def hierarchical_summarize(text: str, chunk_size: int = 4000, model="gpt-4o-mini"):
    """Summarize very long documents by summarizing chunks, then summarizing summaries"""
    enc = tiktoken.encoding_for_model(model)
    tokens = enc.encode(text)
    
    if len(tokens) <= chunk_size:
        return text  # Fits in context, no summarization needed
    
    # Split into chunks
    chunks = []
    for i in range(0, len(tokens), chunk_size):
        chunk_tokens = tokens[i:i + chunk_size]
        chunk_text = enc.decode(chunk_tokens)
        chunks.append(chunk_text)
    
    print(f"Summarizing {len(chunks)} chunks...")
    
    # Summarize each chunk
    chunk_summaries = []
    for i, chunk in enumerate(chunks):
        response = client.chat.completions.create(
            model=model,
            messages=[
                {"role": "system", "content": "Summarize the key points from this text section. Be concise but complete."},
                {"role": "user", "content": chunk}
            ],
            max_tokens=500
        )
        chunk_summaries.append(response.choices[0].message.content)
        print(f"  Chunk {i+1}/{len(chunks)} summarized")
    
    # Combine summaries
    combined = "\n\n".join(f"Section {i+1} Summary:\n{s}" for i, s in enumerate(chunk_summaries))
    
    # If combined summaries still too long, summarize again recursively
    if len(enc.encode(combined)) > chunk_size * 2:
        return hierarchical_summarize(combined, chunk_size, model)
    
    return combined

Strategy 3: Smart Truncation

When you must truncate, be strategic about what to keep:

def smart_truncate_conversation(messages: list, max_tokens: int, enc) -> list:
    """Keep system prompt + recent messages; summarize the gap"""
    
    system_msgs = [m for m in messages if m["role"] == "system"]
    chat_msgs = [m for m in messages if m["role"] != "system"]
    
    # Always keep system prompt + last N messages
    system_tokens = sum(len(enc.encode(m["content"])) for m in system_msgs)
    available = max_tokens - system_tokens - 500  # Leave 500 for generation
    
    # Work backward from most recent
    kept_messages = []
    for msg in reversed(chat_msgs):
        msg_tokens = len(enc.encode(msg["content"]))
        if available - msg_tokens > 0:
            kept_messages.insert(0, msg)
            available -= msg_tokens
        else:
            break
    
    # If we dropped messages, add a summary placeholder
    dropped = len(chat_msgs) - len(kept_messages)
    if dropped > 0:
        summary_msg = {
            "role": "system",
            "content": f"[Note: {dropped} earlier messages were omitted due to context length]"
        }
        kept_messages.insert(0, summary_msg)
    
    return system_msgs + kept_messages

The "Lost in the Middle" Problem

Research by Liu et al. (2023) demonstrated that model performance degrades when relevant information is in the middle of a long context:

Performance on multi-document QA with relevant document at different positions:
(Lower position index = earlier in context)

Position 1 (first): 75% accuracy
Position 5 (middle): 58% accuracy  ← significant drop
Position 10 (last): 74% accuracy

The model "focuses" on beginning and end; middle information is less attended to.

Mitigation Strategies

def optimize_context_placement(
    system_instructions: str,
    key_facts: str,         # Most important information
    supporting_docs: str,   # Secondary information
    user_query: str
) -> list:
    """Place most important information at start and end"""
    
    messages = [
        {
            "role": "system",
            "content": f"{system_instructions}\n\n# CRITICAL INFORMATION:\n{key_facts}"
        },
        {
            "role": "user", 
            "content": f"Background documents:\n{supporting_docs}\n\n---\n\n"
                      f"IMPORTANT REMINDER - Key facts to use: {key_facts}\n\n"
                      f"Question: {user_query}"
        }
    ]
    return messages

# Key facts repeated at start (system) and end (user message)
# Supporting docs in the middle where they matter less for retrieval

Long Context vs RAG Decision Guide

Document size < 50 pages AND needs full-document synthesis:
→ Long context is simpler and often better

Document size > 50 pages AND looking up specific facts:
→ RAG is more accurate and cost-effective

Large knowledge base (1000s of documents):
→ RAG required (no context window can hold everything)

Real-time data or frequently updated knowledge:
→ RAG (update document store without retraining)

Need citations to specific sources:
→ RAG provides exact source attribution

Creative or reasoning task needing full context:
→ Long context (RAG loses surrounding context)

High-volume production queries:
→ RAG (shorter prompts = lower cost per query)

Context Window Costs

Context window size directly impacts cost — most models charge per token:

Example: Analyzing a 50-page document (35K tokens)

GPT-4o:
- Input cost: 35K × $5/1M = $0.175 per analysis
- At 1,000 analyses/day: $175/day

GPT-4o mini:
- Input cost: 35K × $0.15/1M = $0.00525 per analysis  
- At 1,000 analyses/day: $5.25/day

RAG approach (chunk to 4K retrieved context):
- Input cost: 4K × $0.15/1M = $0.0006 per query
- At 1,000 queries/day: $0.60/day

For cost-sensitive applications: RAG wins decisively.
For quality on complex cross-document analysis: long context may justify cost.

Conclusion

The context window is one of the most practically important LLM characteristics. Larger windows open new use cases — entire codebases, full books, extended conversations — but don't automatically produce better results. The "lost in the middle" problem means naive long-context use can perform worse than thoughtful chunking.

The practical skill: know when long context is worth the cost, structure your prompts to put critical information at the start and end, and use RAG when you need cost efficiency or scale beyond any context window.

For building RAG systems that work with large knowledge bases, see our RAG guide. For comparing models by context window and other capabilities, see our GPT-4 vs Claude vs Gemini comparison.

Frequently Asked Questions

What is a context window in LLMs?

The maximum tokens an LLM processes in one pass — both input and output combined. The model can only see and reason about text within this window. Tokens outside the window are invisible. Think of it as the model's working memory.

How many tokens is 1000 words?

Approximately 1,300-1,500 tokens for average English text. Rough guide: 1 page ≈ 650-750 tokens. 128K token window ≈ 96,000 words ≈ ~300 pages. Use tiktoken to count precisely.

Why does context window size matter for practical use?

Determines document length you can process, conversation history length, and code analysis scope. Key tradeoff: larger context increases compute cost and triggers the "lost in the middle" problem. RAG often outperforms naive long-context for specific fact lookup.

What is the "lost in the middle" problem?

Models perform better on information at the beginning or end of the context window than in the middle. Mitigation: place critical information at start and end; use RAG to retrieve and position relevant content prominently; split very long documents and analyze sections separately.

When should I use long context vs RAG for document analysis?

Long context: full document synthesis under ~200 pages, creative/reasoning tasks needing full context. RAG: large knowledge bases (1000s of docs), specific fact lookup, high-volume production queries (cost matters), real-time or frequently updated data.