robots.txt + Sitemap

Create a clean robots.txt that points crawlers at your sitemap, plus a sitemap.xml starter you can grow. Good hygiene for any new site — it tells search engines what to index and where to look.

Try it now

Worked example

# robots.txt — put at your site root. User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml # ---- sitemap.xml starter — add one <url> per page ---- <?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url><loc>https://example.com/</loc></url> </urlset>

Learn the method

Inputs

Your domain
example.com
Paths to disallow (comma-sep)

Frequently asked

Do I need both files?

robots.txt is recommended and points to your sitemap; the sitemap lists your pages. Both help search engines crawl you efficiently.

How do I add more pages?

Copy the <url><loc>…</loc></url> line for each page in your sitemap.xml.