Robots.txt Generator

What Is a Robots.txt File?

A robots.txt file is a plain text file placed in the root directory of a website that instructs web crawlers which pages they are allowed to access and which they should skip. It is the primary mechanism through which website owners communicate crawling preferences to search engine bots like Googlebot, Bingbot, and hundreds of other automated crawlers.

The file follows the Robots Exclusion Standard, a protocol established in 1994 and still widely used by all major search engines. The standard defines three key directives: User-agent (which bot the rule applies to), Disallow (paths the bot should not crawl), and Allow (paths that override a broader Disallow rule). The wildcard user-agent (*) applies rules to all bots that have not been given specific instructions in a separate block.

It is important to understand what robots.txt does not do. It prevents crawling, not indexing. A page blocked by robots.txt will not be crawled, but if other pages link to it, Google may still index the URL based on those links without seeing the page content. To prevent indexing, a noindex meta tag or X-Robots-Tag HTTP header is required on the page itself. The difference matters: robots.txt manages crawl budget and access, while noindex manages whether a page appears in search results.

According to Google's robots.txt documentation, the file is fetched by Googlebot before it begins crawling a site and is re-fetched periodically as the site is re-crawled. The RFC 9309 standard for robots.txt published in 2022 formalises the protocol used by all major search engines today.

Crawl budget is the primary practical reason to maintain a well-configured robots.txt file. Large sites with thousands of pages need search engine crawlers to spend their limited crawl capacity on the pages that matter: product pages, content articles, and key service pages. If crawlers spend time on session-parameter URLs, internal search result pages, duplicate filtered views, and admin paths, they have less capacity available for new and updated content pages that need to be indexed promptly.

How to Use the Robots.txt Generator

1. Configure your first user-agent block. The default is the wildcard (*) which applies to all crawlers. Change the user-agent field to target a specific bot like Googlebot if you need bot-specific rules.
2. Add Disallow paths for folders or pages you want to block. Common examples are /admin/, /private/, /api/, /search/, and /cart/.
3. Add Allow paths if you need to override a broad disallow rule. For example, if you disallow /api/ but want to allow /api/public/, add /api/public/ as an allow rule.
4. Click Add user-agent block to create additional rules for specific crawlers, such as stricter rules for aggressive crawlers or permissions for a specific content syndication partner.
5. Enter your sitemap URL in the Sitemaps section. This is not required but helps search engines find your sitemap automatically.
6. Set a crawl delay in seconds if you want to reduce crawl frequency. This is applied to the wildcard block and can help reduce server load from heavy crawl activity.
7. Copy the generated output and save it as a plain text file named robots.txt. Upload it to the root of your website so it is accessible at yourdomain.com/robots.txt.
8. Test the file using Google Search Console's robots.txt tester after uploading to verify the rules behave as expected.

Common Robots.txt Directives Compared

When to Use the Robots.txt Generator

The Developer Setting Up a New Website

A developer is launching a new e-commerce site with a staging subdirectory at /staging/, an admin panel at /admin/, an internal search results path at /search/, and a checkout flow at /checkout/. Before the site goes live, she uses the generator to create a robots.txt that disallows all four paths for all crawlers, adds the production sitemap URL, and keeps the root domain fully crawlable. The file is uploaded to the root directory as part of the pre-launch checklist, ensuring that no internal pages are accidentally crawled and indexed before the launch is complete.

The SEO Manager Recovering Crawl Budget on a Large Site

An SEO manager at a retailer with 50,000 product pages notices in Google Search Console that the crawl budget is being consumed heavily by filtered URL variants: /products/?colour=red, /products/?size=large, and hundreds of similar combinations. These filtered pages are duplicates of canonical category pages and do not need to be indexed. She uses the generator to add a disallow rule for URLs containing query parameters by blocking /products/? and updates the robots.txt. Within eight weeks, Google's crawl of the site focuses significantly more heavily on unvisited product pages, and the indexation rate for new products improves markedly.

The Agency Configuring Bot-Specific Rules for a Media Client

An agency is managing a news site that wants to allow Googlebot to crawl all content but wants to restrict an aggressive content aggregation bot, Semrushbot, from accessing the full article archive. The agency uses the generator to create two separate user-agent blocks: one for * with standard rules, and a separate block for SemrushBot with a disallow rule for /archive/. The different block structure ensures that Google is unaffected while the specific aggressive crawler is limited to the current content area of the site.

The In-House SEO Auditing a Misconfigured Robots.txt After a Migration

An in-house SEO discovers after a site migration that the staging server's robots.txt, which had Disallow: / for all crawlers, was accidentally deployed to the production server. The entire site has been blocked from crawling for three days. She uses the generator to immediately create a correct production robots.txt with only the intended disallow rules and none of the blanket staging block. The correct file is deployed within minutes and Google is notified via the Search Console URL inspection tool to recrawl the homepage immediately.

Advanced Robots.txt Configuration Tips

Targeting specific bots with dedicated user-agent blocks allows you to apply different crawling policies to different automated programs. Googlebot, Bingbot, DuckDuckBot, Slurp (Yahoo), and Facebot (Facebook) are the major named crawlers that most sites interact with. If you are running a media site and want to specifically control how Google News indexes your content, a separate Googlebot-News block lets you disallow the news bot from older archive sections while keeping all content accessible to the main Googlebot crawler. This level of granularity is only possible through named user-agent blocks.

Path specificity matters for robots.txt rules. When you write Disallow: /products, you block /products/ as well as any URL that starts with /products including /products-reviews/, /products-info/, and similar paths you may not intend to block. Using Disallow: /products/ with a trailing slash is more precise: it blocks the directory and all its children but would not affect a page literally named /products if such a page existed. Always check the paths you intend to block against your actual URL structure to avoid unintended collateral blocking.

Query parameter blocking is one of the most practically valuable uses of robots.txt for large sites. E-commerce sites with faceted navigation frequently generate thousands of duplicate or near-duplicate URLs through filter combinations such as /category/?color=red&size=large&brand=xyz. Each variation is technically a unique URL but contains no unique content. Blocking these parameter URLs with a rule like Disallow: /category/? prevents crawlers from spending time on these low-value pages. Combine this with canonical tags on the filtered pages to ensure any ranking signals those pages receive consolidate onto the canonical version.

Testing your robots.txt rules before deployment is essential and often skipped. The Google Search Console robots.txt tester allows you to enter specific URL patterns and see whether they would be allowed or blocked by your current live robots.txt. If you do not have access to Search Console for the domain you are configuring, you can manually parse the file by reading the user-agent blocks in order: find the most specific block that matches the user-agent you are testing, then check the Allow and Disallow rules within that block from most specific to least specific. A more specific path rule overrides a broader one, regardless of the order they appear in the file.

Documenting changes to your robots.txt file with version history is a practice that saves significant debugging time when issues arise. A simple version comment at the top of the file noting the date and reason for each change gives you a reference when investigating why a particular section of your site stopped being crawled or started receiving unexpected crawl traffic. Maintaining the robots.txt file in your version control system alongside your codebase ensures that changes are tracked, reviewed, and deployable through your normal release process rather than being edited directly on the server where changes are invisible to the rest of the team.

Common Robots.txt Mistakes to Avoid

Blocking CSS and JavaScript files is one of the most damaging robots.txt errors on modern sites. Google renders pages before indexing them, which means it needs to be able to fetch the CSS and JS files that control what a page looks like and how it functions. A robots.txt rule that blocks /wp-content/ or /assets/ can prevent Google from rendering pages correctly, which may suppress rankings for affected pages. Never block static asset directories unless you have a specific and deliberate reason to do so.

Using robots.txt to hide sensitive content is a security misunderstanding. The robots.txt file is publicly readable by anyone, which means listing your /admin/, /internal/, or /private/ paths in a disallow rule explicitly tells anyone who reads the file where your sensitive directories are located. Robots.txt controls crawling; it does not provide access control. Use server-level authentication, firewalls, or access controls to protect sensitive content.

Forgetting to remove the staging robots.txt when deploying to production is a common and high-impact error. Staging environments typically have Disallow: / to prevent search engines from indexing the pre-launch site. If this file is copied to production during deployment, the entire site becomes uncrawlable in hours. Always verify the production robots.txt after any deployment by fetching yourdomain.com/robots.txt directly.

Assuming crawl-delay works for Googlebot leads to frustrated expectations. Google has stated that Googlebot does not respect the crawl-delay directive. To slow Googlebot's crawl rate, use the crawl rate settings in Google Search Console under Site Settings. The crawl-delay directive may still be useful for non-Google crawlers that do respect it.