1. Requirements & Core Workflows
A cloud storage service like Google Drive or Dropbox allows users to store, sync, and share files across desktop, web, and mobile devices. Syncing multi-gigabyte files over unstable home internet requires breaking files into smaller chunks.
If a user modifies only 2 paragraphs of a 500-page PDF document, re-uploading the entire 50MB file wastes mobile data and server bandwidth. Delta sync ensures only modified chunks are transmitted.
2. Variable-Length Chunking with Rabin Fingerprints
Fixed-size chunking (e.g. dividing files into static 4MB blocks) fails when a user inserts bytes at the beginning of a file, shifting all subsequent chunk boundaries and invalidating the entire chunk cache.
Content-Defined Chunking (CDC) using rolling Rabin fingerprints scans file bytes and splits chunks at boundary markers determined by content patterns. Inserting text only alters 1 or 2 neighboring chunks, preserving the hashes of all other blocks.
3. Content-Addressed Storage (CAS) & Global Deduplication
Each chunk is hashed using SHA-256 to produce a unique 256-bit identifier. Chunks are stored in S3/Ceph object storage indexed by their hash value. If 1,000 users upload the same operating system installer image, the system stores the chunks once globally, saving petabytes of storage.
4. Metadata Engine & Conflict Resolution
File hierarchy, permissions, and chunk manifests are stored in a distributed relational database (Spanner or PostgreSQL). When two users edit a file offline and reconnect simultaneously, the server detects conflicting version vectors and creates a conflict copy (`filename (conflicted copy)`).
Model this architecture on the HLDForge Canvas
Turn static box diagrams into living, collaborative systems. Drag and drop microservices, Kafka clusters, and cache tiers. Run automated SPOF detection and STRIDE threat modeling in real time.
Written by HLDForge Architecture Team
Principal Storage Systems Architect
The HLDForge Architecture Team publishes practical engineering guides, real-world case studies, and high-concurrency blueprints to help engineers build scalable and fault-tolerant software.