Design a Web Crawler: System Design Interview Answer (2026)

Web crawler system design is the process of building a distributed service that starts from seed URLs, fetches pages, extracts links, and repeats, while respecting robots.txt, never overloading any single website, and avoiding duplicate work. A strong interview answer centers on three things: the URL frontier (what to fetch next and when), deduplication (of URLs and of content), and a capacity estimate that justifies how many machines you need. Everything else is supporting detail.

Key Takeaways

• At 1 billion pages per month, a crawler needs about 400 fetches per second on average and roughly 800 at peak, which is a modest number of machines if fetching is asynchronous.
• The URL frontier is the heart of the design: front queues handle priority, back queues give each host its own lane, and a heap of "next allowed fetch time" enforces politeness.
• Use a bloom filter for "seen URL" checks (about 12 GB for 10 billion URLs at 1% false positives) and SimHash fingerprints for near-duplicate content.
• Follow RFC 9309 for robots.txt: cache per host for up to 24 hours, treat 4xx as allow-all and 5xx as disallow-all.
• Traps and failures are where senior candidates separate themselves: per-host budgets, URL normalization, depth limits, and idempotent, checkpointed workers.

How should you open the web crawler interview?

Open by clarifying requirements, because "design a web crawler" can mean a search indexer, a price monitor, or an archive. Spend three to five minutes here. The framework from our guide on how to ace a system design interview applies, but crawlers have a few questions you should always ask.

Questions worth asking:

1. What is the purpose? Search indexing, archiving, or data mining changes what you store and how often you recrawl.
2. What content types? Assume HTML only unless told otherwise. Images and PDFs multiply storage.
3. What scale? A common prompt is 1 billion pages per month.
4. Do we recrawl? Freshness matters for news sites and barely matters for static documentation.
5. How long do we keep pages? Assume five years for archive-style storage.

State the requirements back. Functional: given seed URLs, fetch pages, extract and enqueue links, store content, recrawl on a schedule. Non-functional: scalable to billions of pages, polite to every host, resilient to malformed pages and hostile servers, and extensible so new content types or processors can be plugged in later.

What is the capacity estimate for 1 billion pages per month?

The estimate turns a vague prompt into concrete hardware. Work it out loud, state each assumption, and round aggressively. The table below uses an assumed average of 100 KB of HTML per page.

The 100 KB figure is an interview assumption, not a measurement. Real pages vary widely, so say that out loud and offer to rerun the math with the interviewer's number.

Now convert the rate to machines. A fetch takes somewhere between a few hundred milliseconds and a few seconds when you include DNS, TCP and TLS setup, and slow servers. If the average is about one second, you need roughly 800 requests in flight at peak. One machine running asynchronous I/O can comfortably hold hundreds to low thousands of open connections, so the fetch layer is a handful of machines plus headroom. The real bottlenecks are politeness (you cannot fetch faster from any single host) and parsing CPU, not network.

What does the high-level architecture look like?

The high-level design is a loop. URLs flow from the frontier to fetchers, pages flow to parsers, and new links flow back into the frontier after filtering. Draw it in order:

1. Seed URLs enter the URL frontier.
2. URL frontier picks the next URL respecting priority and politeness.
3. DNS resolver (with a local cache) turns hostnames into IPs.
4. Fetchers download pages, honoring robots.txt and timeouts.
5. Content store saves the raw page (object storage) and metadata (a wide-column or key-value store).
6. Content dedup checks a fingerprint and drops duplicates before parsing.
7. Parser / link extractor pulls out links and normalizes them.
8. URL filter applies robots rules, blocklists, and trap heuristics.
9. URL dedup (bloom filter plus a durable URL store) drops already-seen URLs.
10. Surviving URLs go back into the URL frontier.

Connect stages with durable queues (Kafka or a similar log) so each stage scales independently and a crash in the parser does not lose fetched pages. This is the same decoupling pattern you would use in a notification system design, and naming the parallel shows the interviewer you recognize reusable patterns.

How does the URL frontier handle priority and politeness?

The URL frontier is a two-stage queue system: front queues decide what is important, back queues decide what is safe to fetch right now. This split comes from the Mercator crawler design and is described in detail in Manning, Raghavan, and Schütze's Introduction to Information Retrieval. Going this deep is the single best way to stand out on this question.

Front queues: priority

A prioritizer assigns each incoming URL a score and places it in one of F front queues, say F = 10, with queue 1 being the highest priority. Inputs to the score include:

• PageRank-style link importance or the host's historical quality
• How often the page has changed in past crawls (frequently changing pages get recrawled sooner)
• Depth from the seed (shallower usually matters more)

A biased selector pulls from front queues, preferring high-priority queues but still draining low-priority ones so nothing starves.

Back queues: politeness

Politeness means a crawler never sends requests to one host faster than that host can handle. The design that makes this simple is the invariant one host per back queue:

• There are B back queues. A common heuristic from the same textbook is about three times as many back queues as fetcher threads, so workers always have a host ready.
• A host-to-queue mapping table records which back queue owns each host.
• When a back queue empties, the router refills it by pulling URLs from the front queues until it finds a host that is not already assigned elsewhere.
• A min-heap holds one entry per back queue, keyed by the earliest time that host may be contacted again.

A fetcher pops the heap, waits until that timestamp if needed, fetches one URL from that back queue, and pushes the queue back onto the heap with a new time. The delay is often set adaptively, for example ten times the duration of the last fetch from that host, so slow servers automatically get gentler treatment.

import heapq
import time

def next_url(heap, back_queues):
    ready_at, queue_id = heapq.heappop(heap)
    wait = ready_at - time.monotonic()
    if wait > 0:
        time.sleep(wait)
    url = back_queues[queue_id].popleft()
    return url, queue_id

def release(heap, queue_id, fetch_seconds, politeness_factor=10):
    delay = max(1.0, fetch_seconds * politeness_factor)
    heapq.heappush(heap, (time.monotonic() + delay, queue_id))

Production code runs this per frontier shard with async waits, but the logic is identical. The heap is effectively a per-host rate limiter, so you can borrow ideas from designing a rate limiter.

Distributing the frontier

At billions of URLs the frontier does not fit on one machine. Shard it by hash of the hostname, not the full URL. Host-based sharding keeps every URL for one site on one node, which means politeness is enforced locally with no cross-node coordination. The trade-off is hot spots for giant sites; handle those by giving very large hosts a dedicated shard and a higher, negotiated crawl rate.

Each shard keeps the tail of each queue in memory and spills the rest to disk, because billions of URLs at around 100 bytes each are hundreds of gigabytes.

How does a crawler respect robots.txt?

A crawler respects robots.txt by fetching the file before crawling any page on a host, caching the parsed rules, and checking every URL against them before it enters a back queue. The rules are now standardized in RFC 9309, published by the IETF in 2022.

Details worth stating in the interview:

• Caching: RFC 9309 says crawlers should not use a cached robots.txt for more than 24 hours unless the file becomes unreachable. Store it in a per-host cache next to the frontier shard.
• Size: parsers must handle at least the first 500 KiB of the file.
• Status codes: a 4xx response generally means no restrictions. A 5xx or network failure means treat the site as fully disallowed. According to Google's robots.txt documentation, Googlebot stops crawling a site for the first 12 hours after a server error on robots.txt, then falls back to the last good copy for up to 30 days.
• Crawl-delay: this directive is not part of RFC 9309 and Google ignores it, but honoring it when present is a cheap politeness win for your own crawler.

Also send a clear User-Agent with a contact URL so site owners can reach you.

How do fetching, parsing, and storage work?

Fetching, parsing, and storage are the throughput layer, and each has one or two decisions that matter.

DNS. DNS lookups can take tens to hundreds of milliseconds and many resolvers are synchronous. Run a local caching resolver per fetcher cluster and prefetch DNS for hosts as they enter back queues. It is a classic, easily missed bottleneck.

Fetching. Use asynchronous HTTP clients with connection reuse per host, strict timeouts (connect, first byte, and total), a maximum response size (a few MB for HTML), and redirect limits. Send conditional requests with If-Modified-Since or ETag on recrawls so unchanged pages cost a 304 instead of a full download.

Parsing. Parsers are CPU-bound, so run them as a separate pool behind a queue. Extract links, resolve relative URLs against the base, and normalize them: lowercase the scheme and host, remove default ports, drop fragments, sort or strip tracking query parameters. If JavaScript rendering is in scope, route only pages that need it to a much smaller pool of headless browsers, because rendering costs orders of magnitude more than plain parsing.

Storage. Split it by access pattern:

Batching pages into large files, as the WARC archive format does, keeps object storage request costs low. Reversed-hostname keys (com.example.www) group a site's pages for per-site scans.

How do you deduplicate URLs and content?

Deduplication happens at two levels, and you should name both without being asked.

URL dedup. After normalization, hash each URL and check a bloom filter. A bloom filter is a bit array with k hash functions that answers "definitely not seen" or "probably seen." The sizing formula is about -ln(p) / (ln 2)^2 bits per item, which is roughly 9.6 bits at a 1% false positive rate, with about 7 hash functions. For 10 billion URLs that is about 12 GB, small enough to shard across a few frontier nodes. A false positive means skipping a URL you never fetched, which is acceptable at a 1% rate for a crawler. If the interviewer wants zero misses, back the filter with an exact store and only consult it on "probably seen" hits.

Content dedup. Many different URLs serve identical content (mirrors, ?ref= variants, printer-friendly pages). Hash the normalized body with a fast hash to catch exact duplicates. For near-duplicates, compute a SimHash fingerprint. Google researchers Manku, Jain, and Das Sarma showed in a 2007 WWW paper that 64-bit SimHash fingerprints with a Hamming distance threshold of 3 detect near-duplicates effectively across a corpus of billions of pages. Searching for fingerprints within 3 bits uses a set of permuted, sorted tables so the lookup stays fast.

Dedup content before parsing links. Otherwise every mirror page dumps the same hundreds of links back into the pipeline, and the URL bloom filter has to absorb them all.

How do you handle crawler traps and failures?

Traps and failures are where interviewers probe for senior-level judgment. A crawler trap is any URL space that generates unlimited unique URLs, whether malicious or accidental.

Common traps and defenses:

The strongest single defense is a per-host budget tied to the host's priority. A trap can then waste one host's allowance but cannot take over the crawl.

For failures, make every stage idempotent and checkpointed:

• Fetcher crashes: URLs are acknowledged in the frontier only after the page is durably written, so a crash leads to a retry, not a loss. Dedup on write makes the retry harmless.
• Retries: use exponential backoff per host, and after repeated failures demote the host's priority instead of retrying forever.
• Frontier node loss: replicate the disk-backed queue log, and rebuild in-memory heaps from it on restart.
• Bad content: malformed HTML, wrong encodings, and huge files should be caught by parser timeouts and size limits so one page cannot stall a worker.

Structured narration matters most in this section. Our guide on how to think out loud in technical interviews has phrasing that works well for walking through failure scenarios.

What follow-up questions should you expect?

Expect two or three follow-ups once the core design is on the board. Short, confident answers here move a mid-level answer to a senior one.

1. How do you decide recrawl frequency? Track change rate per page from past fetches and schedule recrawls in proportion to it. Use conditional GETs so unchanged pages are cheap.
2. How do you crawl JavaScript-heavy sites? Classify pages by whether the raw HTML contains meaningful content, then send only the rest to a headless browser pool with its own lower throughput budget.
3. How do you scale to 10x? The frontier shards by host and the fetch layer is stateless, so add nodes. The bloom filter grows linearly, around 120 GB for 100 billion URLs at 1%, still shardable.
4. How would you prioritize fresh news? Add a fast lane: seeds from RSS feeds and sitemaps go straight to a high-priority front queue with short recrawl intervals.
5. What would you monitor? Fetch rate per shard, error rate by status code, queue depth, robots.txt failures, and duplicate ratio. A rising duplicate ratio is often the first sign of a trap.

Related designs reuse the same building blocks. The key-value and caching ideas overlap with designing a distributed cache, the hashing and ID mapping overlap with the URL shortener design, and the ranking side connects to typeahead autocomplete design.

How do you practice this question so it sticks?

Time yourself through the full answer in 45 minutes, then repeat only the frontier deep dive until you can explain front queues, back queues, and the heap in under five minutes without notes.

Then run it as a mock with a peer and ask for follow-ups you have not rehearsed. Our guide on how to practice system design interviews lays out a weekly plan, and the Google technical interview process guide covers how search-heavy loops grade design rounds.

Strong candidates do not just memorize the diagram. They explain why the frontier shards by host and why a bloom filter is acceptable. Those reasons are what interviewers write down.