1. Why Scale-Up Fails
A single primary database server, even with 128 vCPUs and 1TB of RAM, eventually runs out of IOPS and memory bandwidth under write-heavy workloads. Horizontal sharding splits a large dataset across multiple independent database instances.
2. Modulo Hashing vs Consistent Hashing
Naive modulo hashing (`shard = hash(key) % N`) causes catastrophic data migration: adding or removing a single node invalidates $(N-1)/N$ of all keys (up to 90%+ data churn).
Consistent Hashing maps both keys and database servers onto a circular 360-degree hash ring ($0$ to $2^{32}-1$). A key is assigned to the first server encountered moving clockwise. Adding a new node only redistributes $K/N$ keys from its immediate neighbor.
3. Hotspot Elimination with Virtual Nodes (Vnodes)
With few physical nodes, hash values cluster unevenly, causing severe hotspots where one server receives 60% of the traffic. Virtual nodes (Vnodes) assign each physical machine hundreds of virtual positions across the ring (e.g. 256 vnodes per server).
This distributes data uniformly and enables proportional capacity weighting for heterogenous hardware.
Model this architecture on the HLDForge Canvas
Turn static box diagrams into living, collaborative systems. Drag and drop microservices, Kafka clusters, and cache tiers. Run automated SPOF detection and STRIDE threat modeling in real time.
Written by HLDForge Architecture Team
Lead Database Architect
The HLDForge Architecture Team publishes practical engineering guides, real-world case studies, and high-concurrency blueprints to help engineers build scalable and fault-tolerant software.