AI Agents & Web Crawlers Directory

Apple

Apple's search crawler powering Siri, Spotlight, and Safari Suggestions.

Baidu

Baidu's web crawler serving the dominant search engine in China.

Microsoft

Microsoft Bing's search crawler, also powering Copilot AI answers.

Anthropic

Anthropic's search indexing crawler for Claude's search features.

DuckDuckGo

DuckDuckGo's web crawler for the privacy-focused search engine.

Google

Google's primary web crawler powering the world's largest search engine.

OpenAI

OpenAI's search indexing crawler for ChatGPT Search features.

Yandex

Yandex's web crawler for the leading search engine in Russia and CIS.

Andi

Andi Search's AI-powered crawler providing direct conversational answers.

OpenAI

OpenAI's real-time browsing agent when ChatGPT users request live web content.

Anthropic

Anthropic's real-time web agent for when Claude users browse live content.

Microsoft

Microsoft Copilot's browsing agent for AI-assisted answers.

DuckDuckGo

DuckDuckGo's AI assistant crawler for generating instant answers.

Google

Google's AI assistant crawlers including Safety and Cloud Vertex bots.

Google

Google Gemini's Deep Research agent that performs comprehensive multi-page research.

Mistral

Mistral's real-time citation fetcher for the Le Chat AI assistant.

Perplexity

Perplexity's real-time browsing agent for live content retrieval.

Perplexity

Perplexity AI's search crawler indexing content for its answer engine.

You.com

You.com's AI search engine crawler.

Allen AI

Allen Institute for AI's research crawler for academic AI development.

Amazon

Amazon's web crawler powering Alexa, Amazon search, and AI services.

Apple

Apple's AI training token controlling how Applebot data is used for Apple Intelligence.

ByteDance

ByteDance's web crawler for TikTok AI and LLM training data.

Common Crawl

Common Crawl's open-source web archive used by multiple AI companies for training.

Anthropic

Anthropic's web crawler collecting training data for Claude models.

Cohere

Cohere's web crawler for enterprise AI and language model training.

DeepSeek

DeepSeek's web crawler for their open-source large language models.

Diffbot

Diffbot's AI-powered web scraping and knowledge graph crawler.

Google

Google's AI training token controlling use of Googlebot-crawled content for AI.

OpenAI

OpenAI's training data crawler for GPT models including ChatGPT and GPT-4.

NICT

Japan's NICT research crawler for AI and multilingual data collection.

Meta

Meta's web indexer for improving Meta AI search and knowledge.

Webz.io

Webz.io's data extraction crawler used by AI companies for training data.

Anthropic

Anthropic's Claude agent that controls a real browser to complete tasks.

OpenAI

OpenAI's browser agent that autonomously performs web tasks for users.

Microsoft

Microsoft's browser automation framework for testing and scraping.

Google

Google's headless Chrome automation library commonly used for scraping.

SeleniumHQ

The original browser automation framework used for testing and scraping.

Ahrefs

Ahrefs' SEO crawler building the world's largest backlink database.

DataForSEO

DataForSEO's crawler powering SEO data APIs for agencies and tools.

Moz

Moz's SEO crawler building their Link Explorer and keyword databases.

Majestic

Majestic's SEO crawler building the largest commercial link index.

Semrush

Semrush's SEO crawler for competitive analysis and backlink tracking.

Meta

Meta's suite of crawlers for Facebook, Instagram, and WhatsApp link previews.

ByteDance

ByteDance's crawler for TikTok link previews and embed generation.

Google

Google's ad quality crawler checking landing pages for Google Ads compliance.

SolarWinds

SolarWinds' enterprise website monitoring and performance tracking service.

StatusCake

Website monitoring service tracking uptime and page speed globally.

UptimeRobot

Popular free uptime monitoring service checking site availability.