AI crawler

CCBot

Common Crawl's crawler, which builds an open web archive widely used for research and AI training.

Operator
Common Crawl
Category
AI crawler
Label
Verified
we can confirm the operator

Documentation from Common Crawl

Allow or block CCBot

A request that really comes from Common Crawl is reported as a good bot. This rule allows CCBot by name, and an imitation does not match it.

{
  "name": "Allow CCBot",
  "enabled": true,
  "match": "all",
  "conditions": [
    {
      "field": "botId",
      "op": "eq",
      "value": "ccbot"
    },
    {
      "field": "botResult",
      "op": "eq",
      "value": "good"
    }
  ],
  "action": "allow"
}

Add it in the console rule editor, or start from the rule templates.

Check a request from your server

Send the request your server received to the bot check with your secret key. The answer says whether the visitor is a good bot, a bad bot or not detected, and which entry it matched, for example CCBot.

curl -X POST https://api.fpchecker.com/v1/bot-check \
  -H 'X-Secret-Key: sk_...' \
  -H 'Content-Type: application/json' \
  -d '{"method":"GET","url":"https://yoursite.com/pricing","ip":"203.0.113.7","headers":{"user-agent":"..."}}'

Read the integration guide

Some entries draw on ai.robots.txt (MIT licence). ai.robots.txt on GitHub