Site files and AI crawlers

Files and AI crawlers… in the site menu has two parts. Save keeps both together, and one Undo takes the change back. Publish to send them.

AI crawlers

Choose Allowed or Blocked for each kind of AI crawler:

• Training: AI companies collecting text to train their models.
• AI answers: AI search engines that index pages to quote them in answers.
• On request: an AI reading a page because someone asked it about the page.

Publishing writes your choices into the site's robots.txt, with a Content-Signal line, for the crawlers that read it. If the site has its own robots.txt, that one replaces these choices.

Files

Add file uploads a file the site serves as it is at its own address, such as a verification file, /ads.txt or /.well-known/security.txt. Under Address, give the path it's served at: it starts with "/" and uses letters, digits and . _ ~ - only, and can't be an address the site writes itself (robots.txt, sitemaps, llms.txt, favicons, /_x/).

A site file is text, HTML, CSS, JS, JSON, XML or an image, up to 1 MB, and a site has at most 50. HTML files and scripts download instead of opening, so a file can't run on your site.