Common Crawl Bot
CCBot reads robots.txt, publishes its IP ranges, and can be blocked in two lines. What that block does and does not reach.
CCBot is the crawler Common Crawl uses to build its public archive. It reads robots.txt before fetching, respects Crawl-delay, honours nofollow on links, and identifies itself with a user agent that names the project.
CCBot/2.0 (https://commoncrawl.org/faq/)Older log lines may still show CCBot/1.0 (+https://commoncrawl.org/bot.html). Both belong to the same foundation.
Verifying CCBot in your logs
A user agent string is free to forge, so scrapers routinely pretend to be CCBot to inherit whatever allowances a site grants it. Common Crawl publishes the IP ranges the real crawler fetches from, which turns verification into a subnet check against your access logs.
As of 11 August 2026 the published ranges are one IPv6 block, 2600:1f28:365:8000::/56, and four IPv4 blocks: 3.41.188.32/29, 18.97.9.168/29, 18.97.14.80/29 and 18.97.14.88/30. Anything claiming to be CCBot from outside those ranges isn’t.
Blocking it
Two lines in robots.txt stop future crawls.
# robots.txt
User-agent: CCBot
Disallow: /Slowing it down instead of blocking it takes one more directive, useful on a small server where the crawl itself is the problem.
User-agent: CCBot
Crawl-delay: 10Check the file for conflicting rules before you add anything. A wildcard User-agent: * block further down doesn’t override a named CCBot group, because crawlers follow the most specific matching group and ignore the rest.
What blocking does and does not do
A robots.txt block keeps CCBot out of crawls that haven’t run yet. It reaches nothing that’s already published.
Every crawl since 2008 stays online in the public S3 bucket, and copies live in derived corpora such as C4 and FineWeb, on academic mirrors, and on the disks of anyone who downloaded them. The foundation’s Opt-Out Ledger, published on 17 September 2025, records legal removal requests from the BBC, the Guardian, the Financial Times, Reuters, the Associated Press & others, plus more than 900 sites entered by the News/Media Alliance. Even those requests apply going forward.
Blocking also costs you presence in the datasets that non-AI projects run on: link graph research, academic search tools and several small independent search engines all read Common Crawl because it’s the only web-scale corpus they can afford.
Making the call
Block CCBot if your business is selling access to your own text, if you’re a publisher acting alongside a licensing position, or if crawler load is measurably hurting a small server.
Leave it open if you want your pages present in open research data and in the corpora that feed retrieval systems, and if the crawl traffic costs you nothing you’d notice.
Decide once, write it down, and audit the file quarterly against the other crawlers you care about. For how the archive is built and what ends up in a crawl, see Common Crawl. For other sources of web-scale data, see Common Crawl alternatives.
Find out what’s holding your site back.
I review your website personally and write a prioritised action plan for your exact site: what to fix first, what to ignore, and what will move rankings.
Keep reading
Common Crawl
A free archive of the public web, published monthly since 2008. What a crawl contains, how to query it, and what it means for your site.
29 Aug 2026 · 8 min read
Common Crawl Alternatives
No public archive replaces it outright. Four routes that cover most of what people use it for.
29 Aug 2026 · 2 min read
Common Crawl Index
Two indexes cover the same crawls. One answers single lookups, the other scales to whole top-level domains.
29 Aug 2026 · 2 min read