Astro 5
Static file: put robots.txt in public/. For a dynamic file (to build the sitemap URL from
site), use an endpoint. Astro calls GET at build time in static mode and writes the body.
import type { APIRoute } from 'astro'
export const GET: APIRoute = ({ site }) => { const sitemap = new URL('sitemap-index.xml', site) const body = `User-agent: *Allow: /
User-agent: OAI-SearchBotAllow: /User-agent: Claude-SearchBotAllow: /User-agent: PerplexityBotAllow: /
# Training opt-out. Remove if the owner is happy to be trained on.User-agent: GPTBotDisallow: /User-agent: ClaudeBotDisallow: /User-agent: Google-ExtendedDisallow: /User-agent: Applebot-ExtendedDisallow: /
Sitemap: ${sitemap.href}` return new Response(body, { headers: { 'Content-Type': 'text/plain; charset=utf-8' } })}sitemust be set inastro.config.mjs, otherwisesiteis undefined here and the@astrojs/sitemapintegration refuses to build. The integration writessitemap-index.xmlandsitemap-0.xml.- Do not keep both
public/robots.txtandsrc/pages/robots.txt.ts. - If the host is Cloudflare Pages/Workers, check the AI Crawl Control policy as described in
the main skill. It can add its own
Disallowlines on top of yours.
AEO Toolkit is an open-source instrument by Advance Labs.