An AI crawler is an automated bot that visits your website to read and collect its text, so an AI company can use that content to answer questions or train a model. Named ones include GPTBot from OpenAI, Google-Extended, and ClaudeBot from Anthropic.
For a small business, these bots are how your words reach an AI assistant in the first place. When someone asks ChatGPT or Google’s AI about your kind of service, the reply is built from pages a crawler read earlier. If the crawlers cannot reach your pages, or your site never tells them they are welcome, your business is missing from the material the answer is built from.
You control who is allowed in. Your robots.txt file can name which crawlers may read you, and newer Content Signals let you say what they may do with what they find. This is a real choice: some owners welcome the assistants by name, others hold specific ones back.
I set this up deliberately on every build, welcoming the major assistants by name so they may read and cite your pages. A crawler can only carry your business into an answer if it is let in and can read the page cleanly. See agent readiness for the fuller picture.