Robots.txt Generator

Build a valid robots.txt file to control how search engines crawl your site.

100% Free Forever

Sitemap URL

Crawl delay (optional, seconds)

User-agent

Disallow paths

Allow paths

No allow rules.

Generated robots.txt

User-agent: *
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

Free Robots.txt Generator

Build a valid robots.txt file in seconds. Control which pages of your website Google, Bing and other search engines are allowed to crawl. Upload the downloaded file to the root of your domain (e.g., yourdomain.com/robots.txt).

How to use this tool

1
Add your sitemap URL
Help crawlers find every page on your site by linking to your sitemap.xml.
2
Set User-agent rules
Use '*' for all bots, or target a specific one like 'Googlebot' or 'Bingbot'.
3
Add Disallow paths
List URL paths search engines should NOT crawl (e.g., /admin/, /cart/).
4
Add Allow exceptions
Use Allow to whitelist specific files inside a disallowed folder.
5
Download & upload
Save the robots.txt and upload it to the root folder of your website.

Pro tips

robots.txt blocks crawling, not indexing — use 'noindex' meta tag to keep pages out of search results.
Always include a Sitemap line so bots discover all your URLs.
Test the file with Google Search Console's robots.txt Tester before going live.
Be careful: blocking '/' will hide your entire site from search engines.