AI crawler

VelenPublicWebCrawler

Velen's crawler, which scrapes public pages to build business data sets and machine learning models.

Operator
Velen Crawler
Category
AI crawler
Label
Unknown
identified by its self-description only

Documentation from Velen Crawler

Allow or block VelenPublicWebCrawler

VelenPublicWebCrawler is identified by its self-description only, so it is reported as a bad bot. This rule denies it by name. Switch the action to allow if you trust it.

{
  "name": "Deny VelenPublicWebCrawler",
  "enabled": true,
  "match": "all",
  "conditions": [
    {
      "field": "botId",
      "op": "eq",
      "value": "velenpublicwebcrawler"
    }
  ],
  "action": "deny"
}

Add it in the console rule editor, or start from the rule templates.

Check a request from your server

Send the request your server received to the bot check with your secret key. The answer says whether the visitor is a good bot, a bad bot or not detected, and which entry it matched, for example VelenPublicWebCrawler.

curl -X POST https://api.fpchecker.com/v1/bot-check \
  -H 'X-Secret-Key: sk_...' \
  -H 'Content-Type: application/json' \
  -d '{"method":"GET","url":"https://yoursite.com/pricing","ip":"203.0.113.7","headers":{"user-agent":"..."}}'

Read the integration guide

Some entries draw on ai.robots.txt (MIT licence). ai.robots.txt on GitHub