Common Crawl Index

The Common Crawl index maps a URL to the exact file, byte offset and length where its archived copy sits. Without it, finding one domain inside the 84.69 TiB of WARC data in a single crawl would mean reading all of it.

Two indexes exist over the same data, and they suit opposite jobs.

The CDX API

The URL index server runs pywb at index.commoncrawl.org and answers queries over HTTPS. Each crawl has its own endpoint named after the crawl id, and the full list of collections is published as machine-readable JSON at /collinfo.json.

curl -s "https://index.commoncrawl.org/CC-MAIN-2026-30-index?url=example.com/*&output=json"

Each result line carries the URL, the fetch timestamp, the HTTP status, the MIME type, a content digest, and the three fields that matter for retrieval: filename, offset and length. Those three let you pull one page out of a multi-gigabyte archive with a ranged HTTP request against https://data.commoncrawl.org/.

The server is heavily rate limited, and going too fast returns 503 and can block your IP for a day. That’s covered on the Common Crawl rate limit page.

The columnar index

The columnar index holds the same records in Apache Parquet at s3://commoncrawl/cc-index/table/cc-main/warc/, partitioned by crawl and subset. It carries 26 columns covering URL components, host and registered domain, TLD, MIME type, detected languages, content digest, fetch status & the WARC filename with record offset and length.

Columnar storage is what makes it fast. A query touching three columns reads only those three, so a per-domain count over a whole crawl can scan a few megabytes rather than terabytes. Common Crawl’s own worked example on Athena scanned 2.12 MB and cost under a cent.

SELECT url, fetch_status, warc_filename
FROM ccindex
WHERE crawl = 'CC-MAIN-2026-30'
  AND subset = 'warc'
  AND url_host_registered_domain = 'example.com'

Athena, Apache Spark and DuckDB all read it in place. The data sits in the public commoncrawl bucket in us-east-1 and needs no AWS credentials, so --no-sign-request is enough for the S3 CLI.

Which one to use

The dividing line is simple enough to apply without thinking about it. If a person is waiting for the answer, query the API. If a machine is, query Parquet.

For what the index points at, and how a crawl is packaged into WARC, WAT and WET files, see Common Crawl.