How to Block Claude-SearchBot

Complete guide to blocking Claude-SearchBot (Anthropic) from crawling your website using robots.txt, server configuration, and Switch workflows.

Operated by AnthropicSearch Engines

Should You Block Claude-SearchBot?

Caution: Claude-SearchBot is a search engine crawler. Blocking it will remove your pages from Anthropic's search index, which directly impacts your organic traffic and visibility.

Only block Claude-SearchBot if you intentionally want to de-index your site from Anthropic. Instead, consider using Switch to serve optimized content or manage specific page access.

Blocking Methods

1robots.txt

High for cooperative crawlers

Add a Disallow rule for Claude-SearchBot's user-agent string in your robots.txt file. This is the standard, cooperative method that well-behaved crawlers respect.

2Server-side UA filtering

High

Configure your web server (nginx, Apache, Cloudflare) to reject requests matching Claude-SearchBot's user-agent patterns. This blocks at the network level before your application processes the request.

3Switch Journey Workflows

Highest — granular, real-time control

Create a custom journey in Switch that detects Claude-SearchBot and routes it to a block action, challenge, redirect, or modified content — without touching your server configuration.

robots.txt — Block Claude-SearchBot

Add the following to your robots.txt file (at the root of your domain) to block Claude-SearchBot:

User-agent: Claude-SearchBot
Disallow: /

robots.txt — Allow with Restrictions

Alternatively, allow Claude-SearchBot on most pages while blocking specific directories:

User-agent: Claude-SearchBot
Disallow: /private/
Allow: /

Claude-SearchBot User-Agent Strings

Use these patterns to identify Claude-SearchBot in your server logs or firewall rules:

Claude-SearchBot

Frequently Asked Questions

Does blocking Claude-SearchBot affect my Google search rankings?

No. Blocking Claude-SearchBot does not affect your Google search rankings. Only blocking Googlebot impacts Google Search visibility.

Does Claude-SearchBot respect robots.txt?

Yes, Claude-SearchBot respects robots.txt directives. Adding a Disallow rule for its user-agent will prevent it from crawling blocked paths.

Can I allow Claude-SearchBot on some pages but not others?

Yes. Use robots.txt to disallow specific directories, or use Switch journey workflows for granular page-level control with conditional logic.

Go beyond robots.txt

Switch detects Claude-SearchBot in real-time and lets you build custom journey workflows — block, challenge, redirect, or serve modified content. No server changes required.

Get Started Free
Claude-SearchBot Profile Agents Directory