AI Glossary / AI crawlers (GPTBot and others)

🌐 AI on the web

AI crawlers are bots that AI companies use to collect content from the web: GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot and others. Some gather data for model training, some fetch pages live to answer a specific user question - two different roles with different consequences for publishers. Publishers can manage access in robots.txt, blocking or allowing individual bots. The decision is now a strategic element: blocking protects content from training, but may also cut the site out of AI answers - a channel where more and more people search for information.