Commercial CrawlersActive

ClaudeBot

Anthropic's web crawler collecting training data for Claude models.

Operated by AnthropicOfficial docs

What is ClaudeBot?

ClaudeBot is Anthropic's web crawler for collecting training data for Claude, their family of AI models. It crawls web content at moderate to high rates (around 500 pages/hour on major sites), making it one of the more aggressive AI training crawlers.

ClaudeBot identifies itself with multiple user-agent patterns including "ClaudeBot", "Claude-Web", and "anthropic-ai". Anthropic has documented their crawling practices and provides IP ranges for verification. It respects robots.txt directives.

Anthropic has separate crawlers for different purposes: ClaudeBot for training, Claude-User for real-time browsing, and Claude-SearchBot for search indexing. This separation gives site owners granular control — you can block training while allowing real-time browsing and search.

User-Agent Strings

These are the known user-agent patterns used by ClaudeBot. Use them to identify this crawler in your server logs or configure robots.txt rules.

ClaudeBot
claudebot
Claude-Web
anthropic-ai
Anthropic

robots.txt example:

User-agent: ClaudeBot
Disallow: /private/
Allow: /

Detection on the Switch Network

3

Sessions detected

3

Total events

95%

Avg confidence

First detected: 2/12/2026 · Last seen: 2/25/2026

How to Manage ClaudeBot

1

Block ClaudeBot if you don't want training data collection.

2

Separate from Claude-User and Claude-SearchBot — granular control available.

3

Use Switch to detect ClaudeBot under all its UA variants (ClaudeBot, Claude-Web, anthropic-ai).

4

Monitor crawl rates — ClaudeBot can be aggressive at ~500 pages/hour.

How to block ClaudeBot

Start managing ClaudeBot today

Switch detects, tracks, and lets you build custom journeys for ClaudeBot and 35+ other AI agents and crawlers. Set up in five minutes.

Get Started Free

Related Agents

Claude Computer Use

Browser Agents

Anthropic

Anthropic's Claude agent that controls a real browser to complete tasks.

Claude-SearchBot

Search Engines

Anthropic

Anthropic's search indexing crawler for Claude's search features.

Claude-User

AI Assistants

Anthropic

Anthropic's real-time web agent for when Claude users browse live content.

AI2Bot

Commercial Crawlers

Allen AI

Allen Institute for AI's research crawler for academic AI development.

Amazonbot

Commercial Crawlers

Amazon

Amazon's web crawler powering Alexa, Amazon search, and AI services.

Applebot-Extended

Commercial Crawlers

Apple

Apple's AI training token controlling how Applebot data is used for Apple Intelligence.

Back to Agents Directory