Social Media Backend: 5K Concurrent WebSockets

System Architecture

Hybrid architecture: traditional Django views for REST API + Django Channels for WebSocket connections. Both share the same Django ORM and business logic. Redis serves as the channel layer for WebSocket message routing and as cache for feed data.

Component diagram

• Load Balancer: AWS ALB (HTTP) + NLB (WebSocket) or single ALB with target groups
• Django Workers (REST): Gunicorn workers handling traditional requests (auth, profiles, posts CRUD)
• Daphne/Channels Workers: ASGI servers handling WebSocket lifecycle (connect, disconnect, messages)
• Redis Cluster: Channel layer (pub/sub for WebSocket broadcast) + feed cache + session store
• PostgreSQL: Primary database (users, posts, follows, messages, notifications)
• Celery + Redis: Background tasks (image processing, email, feed fan-out jobs)

Real-time Layer (Django Channels)

Django Channels extends Django to handle WebSockets, HTTP2, and other async protocols. Each WebSocket connection maps to a Consumer instance. Redis channel layer enables cross-process message broadcasting.

Consumer structure

class NotificationConsumer(AsyncWebsocketConsumer):
    async def connect(self):
        self.user = self.scope["user"]
        if self.user.is_anonymous:
            await self.close()
            return
        # Join user-specific group
        self.group_name = f"user_{self.user.id}"
        await self.channel_layer.group_add(self.group_name, self.channel_name)
        await self.accept()

    async def disconnect(self, close_code):
        await self.channel_layer.group_discard(self.group_name, self.channel_name)

    async def send_notification(self, event):
        await self.send(text_data=json.dumps(event["data"]))

Connection scaling

• 5K concurrent connections across 4 Daphne workers (1250 conn/worker)
• Memory per connection: ~2KB (minimal consumer state)
• Heartbeat: 30s ping/pong to detect dead connections
• Graceful shutdown: Drain connections before worker restart (Kubernetes preStop hook)

Feed Generation: Fan-out on Write

Feed fan-out on write: when a user posts, the post ID is pushed to all followers' feed lists in Redis. Read path is O(1) — just fetch the pre-computed feed. Write path is O(followers) but async via Celery.

Why fan-out on write?

Implementation

# On post creation (Celery task)
def fan_out_post(post_id, author_id):
    follower_ids = Follow.objects.filter(following_id=author_id)\
                                 .values_list('follower_id', flat=True)
    pipeline = redis.pipeline()
    for fid in follower_ids:
        pipeline.zadd(f"feed:{fid}", {post_id: time.time()})
        pipeline.zremrangebyrank(f"feed:{fid}", 0, -1001)  # Keep latest 1000
    pipeline.execute()

Celebrity handling

Users with >10K followers are excluded from fan-out. Their posts are fetched at read time via a separate "celebrity posts" key and merged with the user's personal feed.

Real-time Messaging

1:1 and group messaging via WebSocket with message persistence in PostgreSQL. Redis pub/sub routes messages to online recipients; offline recipients get push notifications via Celery.

Message flow

1. Sender sends message via WebSocket → Django Channels consumer
2. Consumer validates, saves to PostgreSQL (Message model)
3. Consumer publishes to Redis channel chat:{conversation_id}
4. All online participants in that conversation receive via WebSocket
5. For offline users: Celery task sends push notification (FCM/APNs)

Message model (simplified)

class Message(models.Model):
    conversation = models.ForeignKey(Conversation, on_delete=models.CASCADE)
    sender = models.ForeignKey(User, on_delete=models.CASCADE)
    content = models.TextField()
    message_type = models.CharField(choices=[('text', 'Text'), ('image', 'Image'), ('file', 'File')])
    read_by = models.ManyToManyField(User, related_name='read_messages', through='MessageRead')
    created_at = models.DateTimeField(auto_now_add=True)

    class Meta:
        indexes = [
            models.Index(fields=['conversation', '-created_at']),  # Pagination
            models.Index(fields=['sender', 'created_at']),
        ]

Data Model Highlights

PostgreSQL for relational integrity: users, follows, posts, conversations, messages, notifications. Proper foreign keys, indexes, and constraints prevent data corruption at scale.

Key tables & indexes

• Follow: Composite unique index on (follower, following); index on following for "followers of X" queries
• Post: Index on (author, -created_at) for profile feed; search_vector (tsvector) for full-text search
• Conversation: Many-to-many via ConversationParticipant; index on (user, -last_message_at)
• Notification: Index on (recipient, -created_at, is_read); partitioned by month for retention

Scaling Metrics

Load test: 5,200 concurrent WebSocket connections, 1,200 msg/sec sustained, <50ms p99 delivery latency. Cost: ~$220/month (4x Daphne, 3x Gunicorn, PostgreSQL db.t3.medium, ElastiCache r6g.large).

Bottlenecks discovered & fixed

• Channel layer backlog: Redis pub/sub queue grew under burst. Fixed by increasing Daphne worker count and adding backpressure (reject new connections when queue > 10K).
• Feed fan-out latency: Large follower batches blocked Celery workers. Fixed by chunking (100 followers/batch) and priority queue for high-priority users.
• PostgreSQL connection exhaustion: Default pool (20) too small for async workers. Increased to 100 per worker + PgBouncer.

Key Architecture Decisions