Check robots.txt.
Avoid blocking yourself.

robots.txt controls crawling, not indexing, and the difference bites

The single most expensive misunderstanding in technical SEO is treating robots.txt as a way to keep pages out of search results. It is not, and using it that way produces the opposite of what you intended.

Disallow tells a crawler not to fetch the URL. It does not tell anyone not to index it. If other pages link to a disallowed URL, search engines can and do index it from those links alone, which is why you occasionally see a result with a real URL, no description, and a note that the page could not be crawled.

The trap that catches almost everyone

Someone wants a page out of the index. They add noindex to the page. Then, to be thorough, they also disallow it in robots.txt.

That combination guarantees the page stays indexed. The noindex instruction lives in the page, and robots.txt has just forbidden the crawler from fetching the page, so the instruction is never read. The rule is straightforward once you see it: to remove a page from the index, it must remain crawlable. Apply noindex, leave it crawlable, wait for the recrawl, and only then consider disallowing it.

Other things worth verifying

The file must be at the root of the host to have any effect, and it is scoped per host, protocol and port, so the http and https versions of your domain are separate files as far as a crawler is concerned. Google caps the file it will process at 500KB. Rules are matched per user-agent group, and a crawler obeys only the most specific group that matches it, so a specific User-agent: Googlebot block means Googlebot ignores your User-agent: * rules entirely rather than combining them.

Blocking CSS and JavaScript is another own goal. Search engines render pages, and a page whose stylesheet is disallowed is evaluated as the unstyled version, which frequently looks broken and mobile-hostile.

How UtilitySEO handles this

The audit reports what your robots.txt is actually doing rather than only whether the file exists, including whether the directives conflict with the indexing signals on the pages themselves.

Further reading

A correct robots.txt is one check inside a larger job. Technical SEO collects the rest of it, crawling, indexing, rendering and redirects, and AI search and answer engines covers the newer question of which AI crawlers respect the file at all.