robots.txt is a small text file at the root of your website that tells automated bots, including search engines and AI crawlers, which parts of your site they may visit and which they should leave alone. It has been part of the web for decades, and every serious crawler checks it first.
For a small business, it is a quiet but real control panel. A misconfigured robots.txt can accidentally hide your whole site from Google, which happens more often than you would expect. Used well, it points crawlers at what matters, keeps them out of pages that should stay private, and, increasingly, states your position on AI.
That last part is newer. Alongside the classic allow and disallow rules, your robots.txt is where you name which AI assistants may read you and, through Content Signals, what they may do with your content. It usually points to your sitemap too, so crawlers find every page.
I set robots.txt deliberately on every build rather than leaving the default in place, so search engines are guided and the AI assistants you want are welcomed by name. A crawler follows the rules only if the rules are actually written. See what a build includes.