News
Working notes on search, crawling and the data behind AI answers. Written from real projects, with sources.
Guide
What Is Common Crawl
A free archive of the public web, published monthly since 2008. What a crawl contains and what it means for your site.
29 Aug 2026 · 8 min read
Reference
Common Crawl Rate Limit
The CDX index server is heavily rate limited. What triggers a block, how long it lasts, and the interface that has no limit.
29 Aug 2026 · 2 min read
Reference
Common Crawl Index
Two indexes cover the same crawls. One answers single lookups, the other scales to whole top-level domains.
29 Aug 2026 · 2 min read
Reference
Common Crawl Bot
CCBot reads robots.txt, publishes its IP ranges, and can be blocked in two lines. What that block does and does not reach.
29 Aug 2026 · 2 min read
Comparison
Common Crawl Alternatives
No public archive replaces it outright. Four routes that cover most of what people use it for.
29 Aug 2026 · 2 min read