robots.txt Generator
Create a robots.txt file to allow or block crawlers and point them to your sitemap. Choose allow-all, block-all, or allow-all-but-block-specific-paths.
Try it now
Worked example
Output: User-agent: *
Allow: /
Sitemap: https://theggnetwork.com/sitemap.xml
Inputs
- Rule
Allow all / Block all / Allow all, block some - Disallow paths (comma)
/admin,/cart,/checkout - Sitemap URL
https://theggnetwork.com/sitemap.xml
Frequently asked
Does robots.txt hide a page?
No — it asks crawlers not to crawl it; use noindex or auth to truly hide content.
Where does robots.txt live?
At the root of your domain, e.g. example.com/robots.txt.