Designing a URL shortener that handles billions of redirects — hash generation, collision handling, caching, and database choices.
A URL shortener looks trivial: take a long URL, return a short code, redirect later. But at bit.ly scale (6 billion clicks/month), every layer matters. Let's design one that actually holds up.
Requirements
- ▸Write: 100M URLs shortened per day (~1200 writes/sec)
- ▸Read: 10:1 read-to-write ratio — 1.2M redirects/sec
- ▸Short codes: 7 characters, URL-safe (a-z, A-Z, 0-9) = 62^7 ≈ 3.5 trillion combinations
- ▸Redirect latency: typescript
const BASE62 = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789' function encode(num: number): string { let result = '' while (num > 0) { result = BASE62[num % 62] + result num = Math.floor(num / 62) } return result.padStart(7, 'a') } // ID 1000000 → "aab_Lxz" (7 chars)
Data Model and Database Choice
The data model is simple: {short_code, long_url, user_id, created_at, expires_at}. The read pattern is a pure key-value lookup by short_code. This is a perfect fit for Cassandra or DynamoDB — single-key reads at massive scale, no joins. For a simpler deployment, PostgreSQL with the short_code as primary key works fine up to ~1B rows.
Caching: The Critical Layer
80% of traffic hits 20% of URLs (power law distribution). A Redis cache in front of the DB with TTL-based expiry handles most redirects without touching the database. Cache-aside pattern: check Redis first, on miss read from DB and populate cache. With 1.2M reads/sec, even a 90% cache hit rate saves 1.08M DB reads per second.
The Full Architecture
- ▸DNS → Load Balancer → Stateless API servers (horizontal scale)
- ▸Write path: API → ID generator → DB write → cache warm
- ▸Read path: API → Redis → (miss) → DB → Redis populate → 301/302 redirect
- ▸Use 301 (permanent) for SEO, 302 (temporary) if you need click analytics
- ▸Analytics: write click events to Kafka → Flink aggregation → ClickHouse for dashboards
- ▸Expiry: background job scans expired rows, deletes from DB and cache