What a robots.txt file actually tells a crawler

A robots.txt file sits at the root of a domain — example.com/robots.txt — and tells well-behaved crawlers which paths they may fetch. It is a request, not an enforcement mechanism: nothing stops a crawler from ignoring it, and it does nothing to hide a page that's already indexed. Once you know those two limits, the syntax itself is almost trivial.

The method

1. Start with a user-agent line: User-agent: * applies the rules that follow to every crawler; naming a specific one (User-agent: Googlebot) applies them only to that bot.
2. Add Allow: or Disallow: lines under that user-agent block. Disallow: / blocks the entire site; Disallow: /admin blocks only paths starting with /admin; an empty Disallow: blocks nothing.
3. Longest-match wins when rules conflict. If Disallow: /blog and Allow: /blog/public both exist, a crawler checks which matched path is longer — /blog/public is longer than /blog, so that one page is allowed even though its parent folder is blocked.
4. Add a Sitemap: line (full URL, not a relative path) so crawlers that respect it know where to find your full URL list.
5. Remember what this file can't do: a disallowed page can still get indexed if something else links to it, just without a crawled description. To truly keep a page out of search results, use a noindex meta tag or require a login — robots.txt only asks crawlers not to request the page, it doesn't hide it.

A worked example

A site wants crawlers to index everything except its admin area and shopping cart, and wants to point them at its sitemap:

User-agent: *
Allow: /
Disallow: /admin
Disallow: /cart

Sitemap: https://example.com/sitemap.xml

Walk the longest-match rule on three real paths:

• /admin/users — matches Disallow: /admin (the rule path /admin is 6 characters) and Allow: / (1 character). The longer match wins, so this path is blocked.
• /cart/checkout — matches Disallow: /cart (rule path /cart is 5 characters) against Allow: / (1 character); blocked.
• /blog/post-1 — matches only Allow: /; nothing disallows it, so it's fetchable.

Nowhere in that file is there a way to make /admin/users disappear from search results if it's already indexed and linked from elsewhere — that's a separate job for a noindex tag on the page itself.

Where people get it wrong

Two mistakes show up constantly. First, treating a blocked path as hidden: a URL can appear in search results with no snippet ("No information is available for this page") because Google saw a link to it without ever being allowed to crawl its content. Second, putting the file somewhere other than the domain root — example.com/blog/robots.txt is never read; it must live at example.com/robots.txt.

The robots.txt generator builds the file from a allow-all, block-all, or allow-with-exceptions choice, so the syntax is never the part you get wrong — only the strategy is left to you.