What Is Common Crawl

Common Crawl is a free archive of the public web, published by a US nonprofit that has been collecting snapshots since 2008. The July 2026 crawl alone holds 2.14 billion pages drawn from 33.2 million registered domains. If your site is crawlable, a copy of it probably sits in there right now.

That archive is the reason the name keeps coming up in conversations about AI. Filtered Common Crawl made up 60% of the training mix for GPT-3, at 410 billion tokens, according to Table 2.2 of the paper OpenAI published in 2020. Almost every open pretraining corpus since then has been distilled from the same source.

This page covers what the archive contains, how it’s packaged, how to query it without getting blocked, and what a site owner can control.

What Common Crawl is

Common Crawl Foundation is a 501(c)(3) nonprofit founded in 2007 by Gil Elbaz, the engineer who built Applied Semantics into the product Google bought & turned into AdSense. Rich Skrenta runs it day to day as executive director. Peter Norvig and Joi Ito have served on its advisory board.

The organisation does one thing: it crawls the web at scale and gives the results away. Funding came almost entirely from the Elbaz Family Foundation Trust until 2023, when Anthropic and OpenAI each donated $250,000.

Its own homepage describes the corpus as over 300 billion pages spanning 15 years, cited in more than 10,000 research papers, growing by 3 to 5 billion new pages a month. Nothing sits behind a login. You can download a terabyte of it this afternoon for the cost of the bandwidth.

Crawl size and schedule

Crawls ship roughly monthly and carry an identifier in the form CC-MAIN-year-week. CC-MAIN-2026-30 ran from 7 to 25 July 2026 and is a fair sample of what a modern crawl looks like.

The 603 million figure is the one worth sitting with. Roughly a quarter of every crawl is URLs the project had never fetched before, which is why a page published last month can appear in a corpus a research team downloads next year.

Crawls aren’t a complete picture of the web and were never meant to be. Common Crawl samples: it follows links, respects robots.txt, and stops. Two consecutive crawls will disagree about which of your pages exist.

WARC, WAT and WET file formats

Each crawl ships as three parallel sets of files, all gzipped, all sharded into roughly 100,000 pieces. They hold the same fetches at three levels of detail.

WARC files hold the complete fetch: the request Common Crawl sent, the response headers your server returned, and the raw body. This is the format the Internet Archive standardised, and it’s what you want if you care about markup, status codes or HTTP headers.

WAT files hold computed metadata as JSON. Every outbound link, every title, every header, without the page body. Link graph work almost always starts here rather than in the WARCs.

WET files hold extracted plain text and nothing else. They’re the smallest of the three at 5.89 TiB compressed for the July 2026 crawl, and they’re what most language model pipelines read first.

The URL index and the columnar index

Downloading 84 TiB to find one domain would be absurd, so the project publishes an index that maps URLs to a file, an offset & a length. Two versions exist, and picking the wrong one is the most common mistake people make.

The CDX server at index.commoncrawl.org answers interactive lookups over HTTPS, and plain HTTP isn’t supported. It runs on pywb, takes a URL pattern plus a crawl identifier, and returns JSON lines. Both interfaces are compared on the Common Crawl index page, and the 503 responses it returns under load on the Common Crawl rate limit page.

curl -s "https://index.commoncrawl.org/CC-MAIN-2026-30-index?url=example.com/*&output=json"

The columnar index is the one for real work. It lives at s3://commoncrawl/cc-index/table/cc-main/warc/ in Apache Parquet, carries 26 columns covering URL parts, MIME type, language, fetch status, WARC filename and record offset, and it’s partitioned by crawl. Athena, Spark and DuckDB all read it directly.

CCBot, robots.txt and IP ranges

The crawler identifies itself as CCBot. Its current user agent string is CCBot/2.0 (https://commoncrawl.org/faq/), and it reads robots.txt before fetching, honours Crawl-delay, and follows nofollow on links.

Blocking it takes two lines.

# robots.txt
User-agent: CCBot
Disallow: /

Common Crawl also publishes the IP ranges CCBot fetches from, which lets you verify a request in your logs rather than trusting a user agent string anyone can forge. As of 11 August 2026 the list covers one IPv6 block, 2600:1f28:365:8000::/56, and four IPv4 blocks starting at 3.41.188.32/29. The decision itself, and what it costs you, is worked through on the Common Crawl bot page.

How the data reaches language models

Common Crawl is raw material, not a training set. Every serious model built on it runs its own filtering pass first, and those passes discard most of what they read.

GPT-3 is the best documented case. The OpenAI paper lists filtered Common Crawl at 410 billion tokens with a 60% weight in the training mix, sampled at 0.44 epochs across 300 billion training tokens, deliberately undersampled relative to curated sources such as Wikipedia and the two books corpora.

Google built C4, the Colossal Clean Crawled Corpus, from a single Common Crawl snapshot in 2019 to train T5. Hugging Face released FineWeb in May 2024, around 15 trillion tokens processed from 96 crawls covering summer 2013 to April 2024, along with a 1.3 trillion token educational subset called FineWeb-Edu.

The practical consequence for a website owner is a lag. A model trained on FineWeb is reading a filtered copy of crawls that ran months or years before the model shipped, so the version of your site an assistant describes may be several rewrites out of date.

The CC-NEWS dataset

News moves faster than a monthly crawl, so the project runs a separate one. CC-NEWS was announced on 4 October 2016, built on StormCrawler rather than the Nutch-based main crawler, and it publishes WARC files daily instead of monthly.

It sits at s3://commoncrawl/crawl-data/CC-NEWS/yyyy/mm/ with a warc.paths.gz manifest per month. Anyone building a news corpus, a media monitoring tool or a dated text collection starts here rather than in CC-MAIN.

Licensing and the opt-out ledger

The terms of use grant a limited, non-transferable, non-exclusive licence to use the service, and they push copyright responsibility onto you. Common Crawl states that crawled content may carry its own separate terms from whoever owns it, and that users must respect third-party rights.

The archive isn’t public domain, and the foundation doesn’t claim otherwise. It reserves the right to remove any crawled content at any time, and it names a copyright agent for infringement claims.

On 17 September 2025 the foundation published its Opt-Out Ledger, a public list of every legal removal request it has received. Site owners are pointed at robots.txt as the routine control, with legal requests handled by mail to the foundation.

The 2025 publisher dispute

On 4 November 2025 The Atlantic published an investigation by Alex Reisner into the archive’s contents. It reported that the corpus held large numbers of paywalled articles from outlets including The Economist, the Los Angeles Times, the Wall Street Journal, the New York Times and The Atlantic itself.

Reisner reported that the foundation had told publishers it respected paywalls while continuing to distribute the material, and that removal requests were not being carried out. Skrenta, quoted throughout the piece, argued that models should be free to read what people can read.

US news publishers followed with a cease and desist letter demanding that the foundation stop scraping and delete the archived material. The Opt-Out Ledger now lists the BBC, the Guardian, the Financial Times, the Washington Post, News Corp, DMG Media, Advance Publications, the Associated Press, Le Monde, Reuters and Hearst Newspapers, plus more than 900 news sites entered by the News/Media Alliance.

One limitation runs through all of it. An opt-out stops future crawls, and it doesn’t reach into corpora already downloaded, mirrored and trained on by third parties around the world.

What to check on your own site

Start with your robots.txt, because it’s the only lever that works without a lawyer. Search it for CCBot, GPTBot, ClaudeBot, Google-Extended and PerplexityBot, and confirm the rules say what you think they say.

Then look up your own domain in the index. A handful of CDX queries will tell you which of your URLs were fetched, what status codes they returned, and whether the crawler is seeing your canonical pages or a mess of parameter variants. Sites frequently discover that most of their crawled URLs are faceted duplicates.

Weigh the block itself on evidence rather than instinct. Blocking CCBot keeps future crawls out of open corpora, and it also removes you from the datasets that academic search tools, link graph research & several small search projects are built on. There are alternatives to Common Crawl if you need web-scale data yourself, and none of them are free in the same way.

If you’d rather not run any of this by hand, that’s a normal thing to hand off. I do it as part of a technical review.