System Design Capstone: Build a URL Shortener at Scale

The Interview Question That Teaches Everything

It is 2008. bit.ly launches as a service for shortening long URLs to compact links. By 2023, bit.ly is handling over 2 billion monthly requests — billions of clicks redirecting users from short links to their destinations. The average click must complete in under 100 milliseconds. The service must never go down.

"Design a URL shortener" is the most common system design interview question at Google, Amazon, Meta, and Uber — not because URL shorteners are particularly important, but because designing one correctly requires you to apply every concept in system design: scale estimation, database choice, caching, load balancing, distributed ID generation, and deployment strategy.

This lesson is a complete walkthrough. By the end, you will have designed a production-grade URL shortener and synthesised everything from all 18 lessons in this course.

Step 1: Requirements Gathering

Before designing anything, clarify requirements. A system design session without requirements is an architect without blueprints.

Functional Requirements

The features the system must provide:

1. Shorten URL: Given a long URL, generate a short code (e.g., bit.ly/abc123).
2. Redirect: Given a short code, redirect the user to the original long URL.
3. Custom alias (optional): User provides their own short code (e.g., bit.ly/my-product).
4. Expiry (optional): Links expire after a set time or date.
5. Analytics (optional): Track click count, geography, referrer, device.

Non-Functional Requirements

The quality attributes the system must meet:

• Availability: 99.99% uptime (52 minutes downtime per year maximum).
• Latency: Redirect must complete in under 100 milliseconds at the 99th percentile.
• Scale: 10 million new URLs created per day; 10 billion redirects per day.
• Durability: Short URLs must never be lost or broken once created.
• Security: Prevent abuse (spam links, malicious URLs).

Step 2: Scale Estimation

Back-of-the-envelope calculations reveal the true design challenges before you commit to an architecture.

Traffic

• Write rate (URL creation): 10 million per day ÷ 86,400 seconds = ~115 writes/second
• Read rate (redirects): 10 billion per day ÷ 86,400 seconds = ~115,000 reads/second
• Read:Write ratio: 100,000 : 115 ≈ 1,000:1. This is an extremely read-heavy system.

Storage

• Each URL mapping: ~500 bytes (short code 10 bytes + long URL ~200 bytes + metadata ~290 bytes)
• 10 million new URLs/day × 365 days/year × 5 years = 18.25 billion URLs
• Storage needed: 18.25 billion × 500 bytes = ~9 TB over 5 years

Bandwidth

• Read traffic: 115,000 requests/second × 500 bytes = ~55 MB/second inbound
• This is manageable — the bottleneck is read latency, not bandwidth.

Key insight from the math: The 1,000:1 read-to-write ratio means the design must optimise aggressively for reads. Caching is not optional — it is the core architectural decision.

Step 3: High-Level Architecture

Component roles:

• Load Balancer: Distributes traffic across API servers. Terminates SSL. Routes based on health checks.
• API Servers (stateless): Handle URL creation and redirect logic. Stateless — any server can handle any request. Scale horizontally by adding more instances.
• Redis Cache: Stores URL mappings in memory. 90%+ of redirect requests should be served from cache, never touching the database.
• Key Generation Service: Pre-generates short codes and stores them in a Redis key pool. Eliminates real-time ID generation from the critical path.
• PostgreSQL Primary: Source of truth for all URL mappings. Handles all writes.
• PostgreSQL Read Replicas: Handle read queries that miss the cache. Horizontally scaled for read capacity.
• Kafka + Analytics Service: Click events are published to Kafka asynchronously. An analytics service consumes them and writes to Cassandra for time-series analytics queries.

Step 4: Deep-Dive Design Decisions

Decision 1: Short Code Generation

A 6-character base62 code (characters a–z, A–Z, 0–9) provides 62^6 = 56.8 billion unique codes — enough for the 18.25 billion URLs over 5 years.

Option A — Hash + truncate: Take a hash of the long URL (MD5, SHA-256), take the first 6 characters. Simple, but:

• Hash collisions require retry logic.
• Different users shortening the same URL get the same short code (may be desirable or not).

Option B — Auto-increment ID + base62 encoding: Use a database auto-increment ID (1, 2, 3...). Convert to base62. ID 1 = "1", ID 62 = "10", ID 3844 = "100". Guaranteed unique, sequential, no collisions.

Option C — Key Generation Service (chosen): Pre-generate 100 million unique 6-character codes and store them in a Redis set. API servers atomically pop keys from the pool on demand. This removes ID generation from the write path entirely — a key is available instantly. Background jobs replenish the pool when it drops below a threshold.

Why KGS wins: No database write is needed during the short code assignment. No collision handling. Ultra-low latency key issuance (Redis SPOP is O(1)).

Decision 2: Database Schema

CREATE TABLE url_mappings (
    short_code   CHAR(8)      PRIMARY KEY,
    long_url     TEXT         NOT NULL,
    user_id      BIGINT,
    created_at   TIMESTAMPTZ  NOT NULL DEFAULT NOW(),
    expires_at   TIMESTAMPTZ,
    is_active    BOOLEAN      NOT NULL DEFAULT TRUE
);

CREATE INDEX idx_short_code ON url_mappings (short_code);
CREATE INDEX idx_user_id ON url_mappings (user_id);

Decision 3: Cache Strategy

Cache-aside with Redis. On redirect:

1. Check Redis for short_code → long_url. (~0.5ms)
2. On hit: return redirect immediately.
3. On miss: query PostgreSQL replica, store in Redis with 24-hour TTL, return redirect.

Target: 90% cache hit rate. With 115,000 redirects/second and 90% hit rate, only 11,500 requests/second hit PostgreSQL — well within the capacity of read replicas.

Cache sizing: Popular links need only 500 bytes per entry. 1 million cached entries = 500 MB. A standard Redis instance handles this easily. The 80/20 rule applies: 20% of URLs receive 80% of traffic. Cache the hot 20%.

Decision 4: HTTP 301 vs 302 Redirect

When a user clicks a short link, the server responds with either a 301 or 302 HTTP status code.

Decision: Use HTTP 302. The primary value proposition of a URL shortener service is analytics. Without click data, the product has limited value. The trade-off: higher server load, which we handle with caching.

Decision 5: Rate Limiting

Without rate limiting, a single bad actor could generate millions of short URLs per minute, consuming the key pool and filling the database with spam.

Rate limiting strategy: Token bucket per API key / IP address.

• Anonymous users: 10 URL creations per minute.
• Authenticated users: 1,000 URL creations per minute.
• Enterprise customers: configurable via API key.

Implementation: Redis with a sliding window counter. INCR and EXPIRE operations atomically enforce the limit with O(1) complexity.

Step 5: Full Architecture Decision Table

What This Capstone Taught You

Designing a URL shortener is a lens through which every system design concept becomes concrete.

The URL shortener handles 10 billion redirects per day. But the same patterns — scale estimation, cache-first reads, async event processing, stateless API servers behind a load balancer — underpin every large-scale system you will ever build: social media feeds, e-commerce platforms, financial systems, and streaming services.

System design is not a collection of tricks. It is a way of thinking: clarify requirements, estimate scale, identify bottlenecks, choose trade-offs deliberately. You have now practised that way of thinking across 18 lessons and one complete design. The next step is applying it to something you are building yourself.