Concurrency Limits and Throttling: Scaling Web Crawlers Responsibly

You probably know how this goes. A deadline is close, the queue looks huge, and somebody bumps the worker count from 8 to 64 to catch up.

It usually works for a few hours. Then response times creep up, 503 errors show up in the logs, and the site owner wants to know why one client is using a third of their capacity.

Making a crawler faster is honestly the easy part. Making it faster in a way site owners put up with takes a bit more thought.

Why total concurrency is a misleading number

Most teams measure throughput as requests per second across the whole fleet, because that’s what the dashboard shows. A target site only sees how many requests hit its own hostname though, and the two numbers can get very far apart.

If you have 200 workers spread across 5,000 domains, each site barely notices you. Point the same 200 workers at one mid-sized retailer during a price check and it can look a lot like an attack.

So the first fix is pretty boring, which is to cap concurrency per host instead of per process. Scrapy calls this CONCURRENT_REQUESTS_PER_DOMAIN (Colly, Crawlee and Apache Nutch have something similar), and 2 to 4 connections per host is a sensible start for sites you don’t know.

How network routing fits in

How your traffic leaves your network matters more than you’d expect. If every request goes out from one address, one noisy job can get the whole office IP flagged.

That’s why a lot of teams move crawler traffic onto a rotating datacenter proxy pool, which keeps research jobs away from company infrastructure and makes testing from different regions much easier. It doesn’t give you a bigger politeness budget, by the way, so if a site is fine with 3 requests per second from your crawler, spreading them over 50 IPs doesn’t suddenly make 150 okay.

Basically the routing layer is there for reliability and location, and the scheduler is still the thing that decides how polite you are.

Throttling that reacts to the server

Fixed delays are a decent first step, but they ignore how the server is actually doing. People have argued about this since the early days, and the politeness policy section on web crawlers mentions gaps anywhere from 1 second to 60 seconds between requests, plus one early crawler that just waited 10 times as long as the last download took.

That idea of adapting to the server still makes a lot of sense. Scrapy’s AutoThrottle extension does something along those lines, where it keeps an eye on latency and slows the crawler down when the server gets slower, so if a page that usually loads in 200 ms starts taking 2 seconds, you back off well before the errors start.

For most mid-sized jobs a reasonable setup is 2 to 4 connections per host and a base delay of 1 or 2 seconds that moves with latency. Add exponential backoff with some random jitter on any 5xx error or timeout, plus a hard hourly limit per host.

Paying attention to what servers tell you

Servers do let you know when they’re struggling, and ignoring that is probably the quickest way to get blocked. Google’s own page on how to reduce Googlebot’s crawl rate says site owners can return 429 or 503 for a short while and Googlebot will slow down across the whole hostname, so a commercial scraper really has no good reason to treat those codes any differently.

A 429 is about as clear as it gets. The MDN reference for 429 Too Many Requests explains that the response can come with a Retry-After header, sometimes set to something like 3600 seconds, and you should wait the full time instead of trying again after 30 seconds because the queue is getting long.

And it helps to fetch robots.txt again every 24 hours or so, since it does change. Respect Crawl-delay when a site sets it, and keep a log of the paths you skipped, because auditors and legal teams tend to like having that record.

Crawlers that keep working

The crawlers that keep running for years usually assume the other site has limited capacity and behave that way. You probably give up around 10 to 20% of raw speed, but you get fewer bans and cleaner data back, and a lot less time goes into fixing broken jobs.

More sites will likely start publishing their rate limits openly, and signals like 429 will probably matter more as bot traffic grows. If you build those signals into your scheduler now, there’s less to sort out later on.

Leave a Comment