Your website has been visited for years by search engine crawlers such as Googlebot. Since the rise of AI, a new group of visitors has joined them: AI crawlers. They gather content on behalf of systems such as ChatGPT, Claude and Perplexity. Understanding who they are and how to deal with them is a new necessity.
What are AI crawlers?
AI crawlers are automated programs that visit websites to gather content for AI systems. Well-known examples are GPTBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot from Perplexity and Google-Extended from Google. Each has its own recognisable name with which it identifies itself on your site.
What do they do with your content?
They read your pages and use that information to feed AI systems, either for training or for retrieving up-to-date information live. If your content is well readable and structured, you increase the chance that it is picked up correctly and later cited.
May you keep them out?
Yes. Through robots.txt you can decide which AI crawlers you allow and which you keep out. Some businesses choose to block everything, out of fear of content misuse. But for most businesses that want to be found in AI answers, it is actually wise to allow these crawlers in.
What is the strategic trade-off?
If you want to be visible in AI answers, AI crawlers must be able to read your content. Block them, and you choose invisibility in that channel. It is a deliberate choice you can make per crawler. We help you make this trade-off and set it up correctly.
