1. Scale, Scope & Constraints
A web search crawler (like Googlebot) indexes 10 billion web pages per month (~3,800 pages per second). The system must respect website politeness (avoiding DoS on target servers), prioritize high-value content, and detect infinite crawl traps and cycles.
Key non-functional requirements include scalability, fault tolerance, extensibility (handling HTML, PDF, images), and strict adherence to robots.txt.
2. URL Frontier & Politeness Queue Architecture
The URL Frontier manages all URLs waiting to be downloaded. To prevent hammering individual domain hosts, the frontier uses a two-tier queue hierarchy: Priority Queues and Politeness Queues.
Priority Queues sort URLs based on PageRank and update frequency. Politeness Queues map URLs to host-specific FIFO queues (one queue per hostname), managed by a delay mechanism that enforces a minimum crawl delay (e.g. 500ms between hits to the same domain).
3. DNS Resolver & Connection Caching
DNS resolution is a primary bottleneck in web crawling. Standard OS DNS lookups are synchronous. The crawler runs an in-memory DNS cache cluster using round-robin DNS prefetching, maintaining active HTTP/2 connection keep-alive pools.
4. Content Fingerprinting with SimHash
Up to 30% of internet web pages are duplicates or mirror sites. To prevent redundant storage, the HTML parser generates a 64-bit SimHash fingerprint for each page. If the Hamming distance between two fingerprints is <= 3, the page is classified as near-duplicate and discarded.
Model this architecture on the HLDForge Canvas
Turn static box diagrams into living, collaborative systems. Drag and drop microservices, Kafka clusters, and cache tiers. Run automated SPOF detection and STRIDE threat modeling in real time.
Written by HLDForge Architecture Team
Lead Search & Indexing Architect
The HLDForge Architecture Team publishes practical engineering guides, real-world case studies, and high-concurrency blueprints to help engineers build scalable and fault-tolerant software.