AI2Bot
Allen Institute for AI's research crawler for academic AI development.
What is AI2Bot?
AI2Bot is the web crawler operated by the Allen Institute for AI (AI2), a nonprofit research institute founded by Paul Allen. AI2 produces open research and models including OLMo and the Semantic Scholar academic search engine.
The crawler collects data for AI research purposes, with a focus on academic and scientific content. AI2's mission is to conduct high-impact AI research for the common good, and their models and datasets are typically released openly.
AI2Bot represents the academic side of AI training data collection. Blocking or allowing it depends on your stance toward open research versus content protection.
User-Agent Strings
These are the known user-agent patterns used by AI2Bot. Use them to identify this crawler in your server logs or configure robots.txt rules.
robots.txt example:
User-agent: AI2Bot Disallow: /private/ Allow: /
How to Manage AI2Bot
AI2 is a nonprofit research institute — data used for open research.
Low crawl volume, minimal server impact.
Consider allowing for sites with academic or research content.
Use Switch to track alongside commercial training crawlers.
Start managing AI2Bot today
Switch detects, tracks, and lets you build custom journeys for AI2Bot and 35+ other AI agents and crawlers. Set up in five minutes.
Get Started FreeRelated Agents
Amazonbot
Commercial CrawlersAmazon
Amazon's web crawler powering Alexa, Amazon search, and AI services.
Applebot-Extended
Commercial CrawlersApple
Apple's AI training token controlling how Applebot data is used for Apple Intelligence.
Bytespider
Commercial CrawlersByteDance
ByteDance's web crawler for TikTok AI and LLM training data.
CCBot
Commercial CrawlersCommon Crawl
Common Crawl's open-source web archive used by multiple AI companies for training.
ClaudeBot
Commercial CrawlersAnthropic
Anthropic's web crawler collecting training data for Claude models.
cohere-ai
Commercial CrawlersCohere
Cohere's web crawler for enterprise AI and language model training.