robots.txt Generator (Free)
Generate a robots.txt file with allow or disallow rules for common search and AI crawlers, plus an optional sitemap line. Check a box for each user-agent, review the preview, then copy or download the file. This is a text generator, not a crawler simulator, and it cannot test how a bot will behave.
Crawler rules
Unchecked bots are omitted. An Allow rule writes Disallow:; a Disallow rule writes Disallow: /. This is not a crawler simulator and does not guarantee how any bot will act.
How to write a robots.txt file without breaking your site
robots.txt is a plain text file served at the root of a domain. It groups rules under a User-agent line. Under each group, an empty Disallow: means no paths are blocked, while Disallow: / asks the crawler to stay out of the whole site. The generator builds one group per checked bot. Choose Allow for crawlers you want to visit and Disallow for those you want to keep out, then add the absolute URL of your sitemap if you have one.
Publish the result as /robots.txt. A common workflow is to allow search engines such as Googlebot and Bingbot, decide separately about AI crawlers such as GPTBot, ClaudeBot, and CCBot, and include your sitemap so crawlers can find it. Keep the file small and specific. Crawlers read the most specific matching rule, so test changes on a staging domain before replacing a working file.
Remember what robots.txt cannot do. It is a voluntary request, not authentication. It does not hide a page from search results if other pages link to it; use a noindex directive or remove the page when you need it out of the index. It does not stop scrapers that ignore the protocol, and user-agent names or vendor policies can change. Verify the current documentation from each crawler owner before you depend on a rule.
Common mistakes: blocking CSS or JavaScript that search engines need to render a page, disallowing a whole directory to hide one file, adding a relative sitemap URL, and publishing a file that blocks everything by accident. This tool generates text only. It does not crawl your site, simulate a bot, or validate the result against a live server. You still need to upload the file and confirm the response.
FAQ
What does a robots.txt file do?
It tells compliant crawlers which paths they may or may not crawl. It is a request, not an access control system, so it does not protect private pages and cannot stop every bot.
Is this robots.txt generator free?
Yes. It runs in your browser with no signup, account, API key, or payment. There is no usage limit.
Does Disallow remove a page from Google?
No. Disallow asks crawlers not to fetch a path. It can still appear in results if other pages link to it. Use a noindex directive or remove the page when you need it out of the index.
Which crawlers can I configure?
The checkboxes cover Googlebot, Bingbot, GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, and anthropic-ai. Crawler names and policies change, so verify the current user-agent documentation before relying on a rule.
Is this a crawler simulator?
No. It only builds robots.txt text. It does not fetch your site, pretend to be a bot, or confirm how a crawler will behave. Test with the crawler owner's official tools.
Where should the file go?
Publish it at the root of the domain as /robots.txt. The sitemap line must be an absolute URL, for example https://example.com/sitemap.xml.