Common Crawl Rate Limit

Common Crawl doesn’t publish a request-per-second number for its URL index. It publishes a behaviour instead: send too many requests in a short window and the CDX server at index.commoncrawl.org starts returning HTTP 503, and your IP can be blocked outright.

The foundation’s own FAQ describes that endpoint as frequently abused and therefore heavily rate limited. Treat 503 as a stop signal rather than an error to retry through.

What triggers it

Four things account for most blocks. Parallel workers hitting the index with no delay between calls. Loops that re-query the same URL pattern instead of caching the answer. Plain HTTP requests, which aren’t supported and fail in ways that look like a network problem. Proxy rotation, which the foundation explicitly asks people not to use.

None of it is exotic. A single unthrottled script that walks a list of 50,000 domains will find the limit within a few minutes.

Staying under it

Use HTTPS, always. The index server doesn’t serve plain HTTP, so http://index.commoncrawl.org produces client errors that get misdiagnosed as rate limiting.

Sleep between calls and stay single-threaded. A short pause per request costs you minutes across a small job and saves you a day of being locked out.

curl -s "https://index.commoncrawl.org/CC-MAIN-2026-30-index?url=example.com/*&output=json"
sleep 1

Cache every response to disk, keyed by URL and crawl id. Most scripts that hit the limit are asking the same question repeatedly across a rerun.

Back off on 503 rather than retrying immediately. Exponential backoff with a ceiling, then stop the job and look at the logs.

The fix that scales

The CDX API is built for interactive lookups, and no amount of politeness turns it into a bulk tool. For anything bigger, the project points people at the columnar index, a Parquet copy of the same data at s3://commoncrawl/cc-index/table/cc-main/warc/.

Athena, Spark and DuckDB read it directly, there’s no shared server to overload, and the same query that would take 50,000 API calls becomes one SQL statement over a partitioned table. Common Crawl’s own index server page recommends it for bulk filtering & aggregation.

The Common Crawl index page covers both interfaces and when to reach for each. For background on how the archive is built and what a crawl contains, see Common Crawl.