Robots.txt Generator

User-agent: *
Allow: /
Disallow: /admin
Disallow: /private

Sitemap: https://example.com/sitemap.xml

What is robots.txt and why does it matter?

A robots.txt file is the first document search engine crawlers like Googlebot, Bingbot and DuckDuckBot request when they visit your website. It lives at the root of your domain (for example, https://example.com/robots.txt) and tells crawlers which areas of your site they are welcome to index and which areas should stay private. Used correctly, robots.txt is a powerful SEO tool — used incorrectly, it can hide your entire site from Google.

Our free Robots.txt Generator removes the guesswork. Pick your user-agent, list the folders you want to block (such as /admin, /cart or/private), add your XML sitemap URL, and download a syntactically correct file in one click. No registration, no email, no upload — everything runs in your browser.

Common robots.txt patterns

• Allow everything: User-agent: * with Disallow: (blank).
• Block one folder: Disallow: /private/
• Block a single page: Disallow: /draft.html
• Hide the entire site: Disallow: / (use only on staging).
• Reference your sitemap: add Sitemap: https://yoursite.com/sitemap.xml at the bottom.

Pair this with our other SEO tools

After generating your robots.txt, build matching XML with the Sitemap Generator, audit on-page tags using the Meta Tag Analyzer, and verify your headings with the Heading Checker. Looking for clean URLs? Convert any title with the Slug Generator.

Best practices and common mistakes

Always test your robots.txt before pushing to production. A misplaced slash or a trailing wildcard can deindex thousands of pages overnight. Keep these rules in mind:

• Robots.txt is public — never use it to hide sensitive data, use authentication instead.
• Blocking a URL in robots.txt does not remove it from Google — use noindex meta tags for that.
• Each subdomain needs its own robots.txt file.
• Always include your sitemap so crawlers can discover new content faster.