robots.txt and Indexing: What Google Crawls and What It Indexes

robots.txt is a small text file with a big effect. One wrong line is enough for Google to stop fetching a single page. At the same time, it is often used for something it can’t do: keeping pages out of the index. Below is what the file actually controls, how to really remove pages from Google and which mistakes we find most often in practice.

Crawling and indexing are two steps

For a page to appear in search results, two things happen:

1. Crawling: a crawler such as Googlebot fetches the page and reads its content.
2. Indexing: Google decides whether the page goes into the index and can therefore show up in search results.

robots.txt only affects step 1. It tells crawlers which paths they may fetch and which not. Whether a page ends up in the index is not something it controls. That is what noindex is for.

According to Google’s documentation, robots.txt is explicitly not a mechanism for keeping a page out of Google. If other pages link to a blocked URL, Google can index it without ever having fetched it. The search result then shows only the address, without a description. In Search Console such a page appears under “Indexed, though blocked by robots.txt”.

How a robots.txt is structured

The file always sits in the root of the domain, at https://your-domain.com/robots.txt. In subfolders it is ignored. It consists of groups: a User-agent line names the crawler, followed by Disallow and Allow rules.

User-agent: *
Disallow: /internal/
Allow: /internal/press/

User-agent: GPTBot
Disallow: /

Sitemap: https://your-domain.com/sitemap-index.xml

A few rules that are often misunderstood:

• User-agent: * applies to all crawlers that have no group of their own. If a crawler has its own group, it follows only that one and ignores the * group completely.
• The longest matching rule wins. In the example, /internal/press/ is allowed even though /internal/ is blocked.
• Paths are case-sensitive. /Internal/ and /internal/ are different paths.
• The sitemap line stands independently of the groups and needs a full URL.
• Not every line is respected. According to its own documentation, Google ignores Crawl-delay, among others, and since 2019 also Noindex in robots.txt.

robots.txt is a request, not an access barrier. Reputable search engines follow it. If you really need to protect something, use password protection on the server.

Really keeping pages out of the index: noindex

If a page should not appear in search results, the noindex belongs on the page itself. There are two ways:

As a meta tag in the HTML head:

<meta name="robots" content="noindex">

As an HTTP header, useful for PDFs and other files without an HTML head:

X-Robots-Tag: noindex

The crucial point: Google must be allowed to fetch the page to see the noindex. If the URL is also blocked in robots.txt, the crawler never reads the instruction. The page can stay in the index even though it carries a noindex. According to Google’s documentation, a noindex on a page blocked by robots.txt is simply not found and therefore ignored.

The rule of thumb:

What belongs in robots.txt

robots.txt is used well when it spares crawlers work that has no value. Typical candidates:

• Internal search results, such as /?s= in WordPress. Every search query creates a new URL with thin content.
• Filter and sort parameters in shops, such as ?orderby= or combinations of colour, size and price. These quickly turn into thousands of variants of the same category page.
• Cart, checkout and customer account. These pages are different for every visitor and have no place in search results.

Google understands two wildcards: * stands for any characters, $ for the end of the URL.

User-agent: *
Disallow: /?s=
Disallow: /*?orderby=
Disallow: /cart/
Disallow: /*.pdf$

What doesn’t belong there are pages you merely don’t want to see in Google, such as thank-you pages or the imprint. noindex is the right tool for those, for the reason described above. And be careful with parameters that also produce real content: if you block ?page=, Google may no longer find the posts on page 2 and beyond.

The most common mistakes

Disallow: / from the development environment. On the staging site this is right, so Google doesn’t find the test version. If the file is carried over at launch, it blocks the entire live website. The effect doesn’t show immediately but over days, as pages gradually lose their description and slip down. In WordPress, check Settings → Reading: the option “Discourage search engines from indexing this site” likes to stay switched on after a move.

noindex and Disallow combined. Trying to secure a page twice achieves the opposite, see above. Let the noindex take effect first, meaning the page gets crawled and drops out of the index. A Disallow is then usually unnecessary.

CSS and JavaScript blocked. Older robots.txt files often block folders such as /wp-includes/ or /assets/. But Google renders pages like a browser. Without stylesheets and scripts it sees a broken page and may misjudge layout and content.

Missing sitemap line. Not a serious mistake, but a missed opportunity. The line helps search engines and other crawlers find all pages, even if you don’t submit the sitemap in every tool separately.

robots.txt answers with a server error. According to Google’s documentation, a 5xx error on robots.txt makes Google stop crawling the whole site at first, because it doesn’t know what is allowed. A 404, on the other hand, is harmless and means “everything allowed”. Whether your file answers cleanly with a 200 you can check with our HTTP status checker.

File too large. According to its specification, Google reads at most 500 KiB, the rest is ignored. For normal websites this doesn’t matter, for automatically generated files with thousands of lines it does.

Checking in Search Console

Search Console shows you how Google sees your robots.txt:

• The robots.txt report (under Settings) shows which version Google last fetched, when that was and whether there were errors. After an urgent fix you can request a recrawl there. According to Google’s documentation, the file is otherwise generally cached for up to 24 hours.
• URL inspection shows for a single address whether crawling is allowed, whether the page is indexed and whether a noindex was detected.
• The “Pages” report under Indexing lists reasons why pages aren’t in the index. The two entries “Blocked by robots.txt” and “Indexed, though blocked by robots.txt” are the most important pointers to mistakes in the file.

After every relaunch, a look at these three places belongs on the checklist, together with checking the redirects as described in Checking HTTP status codes.

AI crawlers in robots.txt

Besides search engines, crawlers from AI providers now read robots.txt too. Many providers distinguish between bots that collect training data and bots that fetch pages for answers in their search. You can allow or block them individually.

Our generator creates the matching groups. You choose per bot or per group whether it is allowed or blocked, and you can add a sitemap URL. It runs entirely in your browser. It only writes the rules for AI crawlers, you carry over your existing rules for all other crawlers into the file yourself.

What trade-off lies behind “block training, allow search” and what it means for visibility in ChatGPT and AI Overviews is covered in our article Visible in ChatGPT & AI Overviews. The generator is also available on its own under robots.txt for AI crawlers.

Short checklist

• The file sits at /robots.txt and answers with status 200 or 404, never with 5xx.
• No Disallow: / for User-agent: * on the live website.
• CSS, JavaScript and images are not blocked.
• Pages that should stay out of Google carry noindex and are not blocked in robots.txt.
• The sitemap line is present and points to a reachable URL.
• After changes: check the robots.txt report in Search Console.

Your pages are disappearing from Google and you can’t find the reason? Our WordPress audit covers indexing, sitemap and robots.txt, or drop us a line.