Static HTML / any host
- Create
robots.txtin the directory the host serves as the site root (public/,dist/,docs/, whatever deploys to/). Use the template in the main skill. - Make sure the host serves it as
text/plainand does not rewrite unknown paths toindex.htmlfor it. A single-page-app fallback rule that catches/robots.txtmakes ittext/html, and the audit then says it is missing. - Per-subdomain: each host needs its own robots.txt (
www.example.com,blog.example.com). Anthropic’s page says to do this for every subdomain you want covered. - If the host or CDN has an “AI bots” toggle (Cloudflare has one), check it matches the robots.txt you just wrote.
Verify: curl -sI https://example.com/robots.txt shows 200 and content-type: text/plain.
AEO Toolkit is an open-source instrument by Advance Labs.