Robots.txt Generator

Control how search engines crawl and index your website

A surprising number of sites I audit either have no robots.txt file or one that's accidentally blocking important pages. This tool walks you through the directives step by step so you don't end up deindexing half your site by mistake.

Basic Configuration

User-agent

Crawl Delay (seconds) Optional delay between requests

Disallow Rules

Paths to Disallow Use * for wildcards, / for directories

Allow Rules

Paths to Allow Override disallow rules for specific paths

Sitemap

Sitemap URL Full URL to your XML sitemap

Quick Templates

Generated Robots.txt

What is Robots.txt?

The robots.txt file tells search engine crawlers which pages or files they can or can't request from your site. It's placed in the root directory of your website.

Best Practices

Use robots.txt to prevent crawling of admin areas, temporary files, and duplicate content. Always include your sitemap URL to help search engines discover your content.

Testing Your File

After uploading robots.txt to your site root, test it using Google Search Console's robots.txt Tester tool to ensure it works as expected.