Generate a robots.txt file for your website. Select which bots to allow or block, configure crawl rules, and add your sitemap URL. Download or copy the result to upload to your site root.

<\!--tool-separator-->

The robots.txt file tells search engines and web crawlers which parts of your site they can access. A properly configured robots.txt improves crawl efficiency, protects private directories, and helps manage the growing volume of AI crawler traffic hitting websites in 2026.