Robots.txt Generator
Free online robots.txt generator. Create and customize robots.txt files to control search engine crawling, manage crawler access, and optimize your site's SEO with an easy visual editor.
What is robots.txt?
Robots.txt is a text file placed at the root of your website that instructs web crawlers (like Googlebot) which pages or directories they can or cannot access. It's part of the Robots Exclusion Protocol (REP).
Where should I place robots.txt?
Place robots.txt at the root of your domain: https://www.example.com/robots.txt. Search engines look for it in this exact location.
What does User-agent mean?
User-agent specifies which crawler the rule applies to. Use * to apply to all crawlers, or specify particular bots like Googlebot, Bingbot, or Baiduspider.
What's the difference between Allow and Disallow?
Allow permits crawlers to access specific paths, while Disallow prevents access. Disallow takes precedence when both rules exist for the same path.
Can robots.txt block access to pages?
Robots.txt provides directives but doesn't enforce access control. Compliant crawlers will respect the rules, but malicious bots may ignore them. Use server-side authentication or password protection for real security.
How do I specify a sitemap in robots.txt?
Add a Sitemap directive: Sitemap: https://www.example.com/sitemap.xml. This helps search engines discover all your important pages.
What is Crawl-delay?
Crawl-delay tells crawlers to wait a specified number of seconds between requests. Example: Crawl-delay: 10 means wait 10 seconds. Note: Google doesn't support this directive.
Should I block admin pages with robots.txt?
Yes, blocking admin areas (/admin/, /wp-admin/) prevents unnecessary crawling and reduces server load. However, blocked pages won't appear in search results even if linked from elsewhere.
Can I have multiple User-agent sections?
Yes, you can have separate rule sets for different crawlers. Each User-agent block should have its own Allow/Disallow rules.
Does robots.txt improve SEO?
Indirectly, yes. By controlling what search engines crawl, you can focus their attention on important pages, prevent duplicate content issues, and reduce server load.
What happens if robots.txt is missing?
If robots.txt doesn't exist, crawlers assume they can access all pages. This is usually fine for small sites but may lead to indexing of unwanted pages on larger sites.
How do I test my robots.txt file?
Use Google Search Console's robots.txt tester, or our built-in validator to check syntax and see how different crawlers will interpret your rules.
Can I block specific file types?
Yes, use Disallow with file extensions: Disallow: /*.pdf$ blocks all PDF files. Use $ to match the end of URLs.
Is robots.txt case-sensitive?
Yes, robots.txt is case-sensitive. /Admin/ and /admin/ are treated as different paths. Always use lowercase for consistency.