Commercial Crawlers Bots & Crawlers

Allen AI

Allen Institute for AI's research crawler for academic AI development.

UA: AI2Bot, ai2bot

Amazon

Amazon's web crawler powering Alexa, Amazon search, and AI services.

UA: Amazonbot

Apple

Apple's AI training token controlling how Applebot data is used for Apple Intelligence.

UA: Applebot-Extended

ByteDance

ByteDance's web crawler for TikTok AI and LLM training data.

UA: Bytespider, bytespider, Bytedance

Common Crawl

Common Crawl's open-source web archive used by multiple AI companies for training.

UA: CCBot, ccbot

Anthropic

Anthropic's web crawler collecting training data for Claude models.

UA: ClaudeBot, claudebot, Claude-Web, anthropic-ai, Anthropic

Cohere

Cohere's web crawler for enterprise AI and language model training.

UA: cohere-ai, CohereBot

DeepSeek

DeepSeek's web crawler for their open-source large language models.

UA: DeepSeekBot, deepseek

Diffbot

Diffbot's AI-powered web scraping and knowledge graph crawler.

UA: Diffbot, diffbot

Google

Google's AI training token controlling use of Googlebot-crawled content for AI.

UA: Google-Extended

OpenAI

OpenAI's training data crawler for GPT models including ChatGPT and GPT-4.

UA: GPTBot, gptbot

NICT

Japan's NICT research crawler for AI and multilingual data collection.

UA: ICC-Crawler

Meta

Meta's web indexer for improving Meta AI search and knowledge.

UA: Meta-WebIndexer, meta-webindexer

Webz.io

Webz.io's data extraction crawler used by AI companies for training data.

UA: webzio