How social feeds are generated at scale — the fanout-on-write vs fanout-on-read trade-off, feed ranking, and celebrity user handling.
Generating a personalized feed for 500 million daily active users is one of the hardest distributed systems problems. The core challenge: when user A posts, how does it appear in the feeds of all A's followers — potentially millions of them — fast?
Two Fundamental Approaches
Fanout-on-write (push model): when a user posts, immediately write that post into every follower's feed cache. Feed reads are instant — just read pre-computed feed. Cost: one write amplifies into millions of writes for celebrity accounts.
Fanout-on-read (pull model): store posts in a user's post table. On feed load, fetch posts from all accounts you follow and merge/sort them. No write amplification, but feed generation is slow and expensive — you're doing a fan-in merge at read time.
The Hybrid Approach (What Twitter Actually Does)
Use fanout-on-write for normal users (typescript
// 1. Pre-computed feed from Redis (fanout-on-write for normal accounts) const cachedFeed = await redis.lrange(`feed:${userId}`, 0, 200) // 2. Merge in celebrity posts (fanout-on-read for high-follower accounts) const celebrities = await getCelebrityFollows(userId) const celebPosts = await Promise.all( celebrities.map(c => getRecentPosts(c.id, limit: 20)) ) }
Feed Ranking
Raw chronological feeds were abandoned years ago. Modern feeds rank by engagement signals: recency, likes, comments, shares, time-spent, relationship strength (how often you interact with this person), content type preference. This ranking runs as a lightweight ML model inference on the merged candidate set before serving.
Data Stores and Infrastructure
- ▸Post storage: Cassandra or DynamoDB — high write throughput, time-series access pattern
- ▸Feed cache: Redis sorted sets (score = timestamp or rank score) per user
- ▸Social graph: Neo4j or a purpose-built graph DB for follower/following relationships
- ▸Fanout workers: Kafka consumers that process post events and write to follower feed caches
- ▸Media: separate CDN pipeline (images/video are not stored with post metadata)
- ▸Feed size cap: keep only last 800 posts in feed cache — older content fetched on-demand