Robots.txt Generator is a free online tool from Tool Hub Zone that creates a robots.txt file for your website. Choose which crawlers and folders to allow or block, add your sitemap address, and copy the finished code. It saves you from writing the rules by hand and making small syntax mistakes. It is part of our SEO tools, and no account is needed.
How to Create a robots.txt File
- Choose the rules you need, such as which folders crawlers should not visit.
- Add the full address of your sitemap.
- Generate the file and copy the code.
- Save it as a plain text file named robots.txt and upload it to the root of your site.
What Is a robots.txt File?
A robots.txt file is a small text file that tells search engine crawlers which parts of your site they may visit. Well-behaved crawlers read it before they request other pages. It must sit in the root of your domain, for example at https://example.com/robots.txt. A copy placed in a subfolder is not read.
What Do the Directives Mean?
| Directive | What it does |
|---|---|
| User-agent | Names the crawler the rules apply to. An asterisk means all crawlers. |
| Disallow | Tells the crawler not to visit a path. |
| Allow | Makes an exception inside a blocked path. |
| Sitemap | Gives the full address of your XML sitemap. |
A Simple Example
This example is typical for a WordPress site. Replace the sitemap address with your own.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap_index.xmlThe first line applies the rules to all crawlers. The next two lines keep crawlers out of the admin area but allow one file that some site features need. The last line points to the sitemap.
Useful Pattern Characters
- * matches any run of characters. For example,
Disallow: /*?sort=blocks URLs that contain a sort parameter. - $ marks the end of a URL. For example,
Disallow: /*.pdf$blocks addresses that end in .pdf.
Paths are case-sensitive, so /Photos/ and /photos/ are different.
Does robots.txt Remove a Page from Google?
No, and this is the most common misunderstanding. Robots.txt controls crawling, not indexing. If other sites link to a blocked page, the address can still appear in search results, without a description. To keep a page out of search results, add a noindex tag to the page, and do not block that page in robots.txt, because crawlers must be able to read the tag. Our Meta Tag Generator can help you create meta tags.
Is robots.txt a Security Tool?
No. The file is public, and anyone can read it by typing the address in a browser. Listing a private folder there can even point people toward it. Some bots also ignore the file. For private content, use a login, a password, or proper server settings.
What Should You Block?
- Admin and login areas that do not need to be crawled
- Internal search result pages that create endless URL combinations
- URLs created by filters and sorting that duplicate other pages
- Cart and checkout pages on a store
What Should You Not Block?
- CSS and JavaScript files. Search engines need them to see your pages as visitors do.
- Pages you want to rank. One careless line can remove important sections from crawling.
- The whole site by mistake. The rule
Disallow: /blocks everything. It is meant for sites that are not ready to be found, and it can seriously hurt a live site.
What About AI Crawlers?
Some AI companies run their own crawlers, such as GPTBot, ClaudeBot, and PerplexityBot, and Google offers a separate control token called Google-Extended. You can allow or block each one with its own User-agent line. If you want your content to be read and possibly quoted by AI tools, make sure these crawlers are not blocked. If you prefer to keep them out, add a Disallow rule for each name. The names and behavior of crawlers can change, so check each company’s current documentation before you rely on a rule.
User-agent: GPTBot
Disallow: /The lines above would block one crawler from the whole site. Remove or change them if you want it to have access.
Common Mistakes
- Placing the file anywhere other than the site root
- Using a different name, such as Robots.txt with a capital R or robot.txt without the s
- Blocking a page and also adding a noindex tag to it, which stops the tag from being read
- Forgetting the sitemap line
- Leaving a test rule on a live site
How to Check Your File
Open yourdomain.com/robots.txt in a browser and make sure the text appears. Then test important pages in Google Search Console to confirm they are not blocked. Google reads only the first 500 KiB of the file, which is far more than most sites need. If you also need a sitemap to list in the file, our XML Sitemap Generator can create one.
If You Use WordPress
WordPress can create a virtual robots.txt file by itself, and many SEO plugins let you edit it from the dashboard. If a physical robots.txt file exists in your site’s root, it takes priority over the virtual one, so check which one your site serves.
Frequently Asked Questions
Is the robots.txt generator free?
Yes. It is free to use, and you do not need to sign up.
Does it work on a phone?
Yes. You can use it in a browser on a phone, tablet, or computer.
Does every site need a robots.txt file?
No. Without one, crawlers can visit everything they find. It is useful when you want to guide them away from certain areas.
Will my site change after I use the tool?
No. The tool only produces text. Nothing changes until you upload the file to your site.
Last updated: October 10, 2026
