Shining Toolbox Shining Toolbox

Robots.txt Generator | Shining Toolbox

Generate robots.txt file online. Control how search engines crawl your website with User-agent, Disallow, and Sitemap directives.

Robots.txt Generator

Create robots.txt file to control search engine crawlers

Rules

Disallow
Allow

Generated robots.txt

# robots.txt generated by Shining Toolbox
# Generated: 2026-01-25T06:23:45.229Z

User-agent: *

About Robots.txt Generator

How to Use the Robots.txt Generator

  1. Choose a preset or start from scratch
  2. Set User-agent - Target all crawlers (*) or specific ones
  3. Add Disallow paths - Pages/directories to block
  4. Add Allow paths - Override Disallow for specific paths
  5. Add Sitemap URL - Help crawlers find your sitemap
  6. Copy and deploy - Place file at your domain root

Common Directives

User-agent

Specifies which crawler the rules apply to:

  • * - All crawlers
  • Googlebot - Google’s crawler
  • Bingbot - Bing’s crawler
  • Yandex - Yandex’s crawler

Disallow

Paths that crawlers should NOT access:

Disallow: /admin/
Disallow: /private/
Disallow: /*.pdf$

Allow

Override Disallow for specific paths:

Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap

Tell crawlers where to find your sitemap:

Sitemap: https://example.com/sitemap.xml

Crawl-delay

Request delay between requests (not all bots honor this):

Crawl-delay: 10

Best Practices

DoDon’t
Block admin areasBlock CSS/JS needed for rendering
Block duplicate contentBlock your entire site accidentally
Include sitemap referenceRely on it for security
Test with Google Search ConsoleBlock important pages
Keep it simpleCreate overly complex rules

Common Presets

Standard Website

User-agent: *
Disallow: /admin/
Disallow: /private/
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

WordPress

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap.xml

Block Everything (Staging Sites)

User-agent: *
Disallow: /

Related Tools

Code Minifier

Minify HTML, CSS, and JSON online. Reduce file size by removing comments, whitespace, and formatting. Free code compression tool.

Diff Checker

Compare two text versions side by side with highlights for additions and deletions. Spot changes in code, configs, or copy instantly with private, in-browser processing.

Hash Generator

Generate MD5, SHA-1, SHA-256, SHA-384, and SHA-512 hashes online. Free cryptographic hash calculator for developers.

Copied to clipboard!