Robots.txt Generator
A robots.txt generator creates a properly formatted robots.txt file that controls how search engine crawlers and bots access your website. Use it to allow or block specific crawlers, set crawl delays, and add sitemap references.
User Agent
Sitemaps
Host Directive
The Host directive tells Yandex the preferred domain for your site. Optional for most sites.
User-agent: * Allow: /
Frequently Asked Questions
What is a robots.txt file?
A robots.txt file is a plain text file placed at the root of your website that tells web crawlers and bots which pages or sections they are allowed or not allowed to access. It follows the Robots Exclusion Protocol and is the first file crawlers check before indexing your site.
Where do I put my robots.txt file?
Your robots.txt file must be placed at the root of your domain, accessible at https://yourdomain.com/robots.txt. It will not work if placed in a subdirectory. Most web hosting platforms and CMS systems provide a way to upload or edit this file directly.
How do I block AI crawlers with robots.txt?
To block AI crawlers, add separate User-agent directives for each AI bot (such as GPTBot, ChatGPT-User, Google-Extended, CCBot, anthropic-ai, and ClaudeBot) followed by Disallow: /. Use our "Block AI Crawlers" preset button above to generate this configuration instantly.
Can robots.txt block all web scraping?
No. The robots.txt file is a voluntary standard - well-behaved crawlers like Googlebot and Bingbot respect it, but malicious scrapers can ignore it entirely. For stronger protection, consider using server-side access controls, rate limiting, or authentication.
How do I test my robots.txt file?
You can test your robots.txt file using Google Search Console's robots.txt Tester, which shows you whether specific URLs are blocked or allowed. You can also validate the syntax by checking that each directive follows the correct format: User-agent, Allow, Disallow, Crawl-delay, and Sitemap.
Need more developer tools?
Learn more about NitroClaw→