Why You Need to Check AI Crawlers
In the past, SEO meant optimizing for search engines. Today, a massive amount of traffic and brand visibility comes from AI models like ChatGPT, Claude, and Perplexity. These models use web crawlers to gather real-time data and train their future models.
The Major AI Crawlers
- GPTBot: OpenAI's crawler used to train ChatGPT. Blocking it means your site won't be used for training.
- ClaudeBot: Anthropic's crawler for the Claude AI model.
- Google-Extended: A specific token for Google's Gemini AI. Note: Blocking this does not block Google Search (Googlebot).
- CCBot: Common Crawl. This is a public dataset used by dozens of AI companies, including early versions of ChatGPT. If you block CCBot, you are blocking a large portion of open-source AI training.
- PerplexityBot: The crawler for Perplexity AI, which generates real-time answers with citations.
Should You Block Them?
It depends entirely on your business model. If you are a publisher who relies on ad revenue, you might want to block AI crawlers because they consume your content without sending traffic back. However, if you want your brand to appear in AI-generated answers, you must allow them to crawl your site.