Search engines

One setting can hide your whole site from Google, and it's easy to leave on when a site goes live from a test version. The opposite matters too: thank-you pages should stay out of Google.

What WP Moo looks at

Every page check reads what the page tells search engines: the robots meta tag in the page's code and the X-Robots-Tag header the server sends with it. "noindex" means "don't list this page", "nofollow" means "don't follow the links on it".

Per page you choose what you expect, under Search engines in the Pages section of Checks & pages:

Found: the page should show up in Google. We report a noindex or nofollow, and a canonical link (the page's "real address" for search engines) that points to another domain.
Hidden: the page should stay out of Google. We report it when noindex or nofollow is missing.
Don't check: we skip these rules for this page.

This setting also decides one WordPress rule: a coming-soon page is only reported on pages set to Found. Don't check keeps whichever of the two the page had.

Why thank-you pages should be hidden

A thank-you page shows after someone orders or sends a form, and that visit often counts as a sale or a lead. If Google lists it, people land there without ordering, and your conversion numbers stop adding up. A page whose address holds words like thank-you, thanks, bedankt or confirmation starts as Hidden when you add it. Every other page starts as Found. When you add a site, its checkout, login page and sitemap start on Don't check: the checkout only shows us an empty cart, and a login page or sitemap is left out of Google on purpose.

What the problems mean

The page is hidden from Google noindex in the robots meta tag or X-Robots-Tag header Important If the whole site has it, untick "Discourage search engines from indexing this site" under Settings › Reading in WordPress. For this page alone, set it to show in search results in its Yoast or Rank Math settings. A noindex in the X-Robots-Tag header comes from the server or a plugin: remove it there, or ask your host.

Google doesn't follow the links on the page nofollow in the robots meta tag or X-Robots-Tag header Notice Set the page's links to be followed in its Yoast or Rank Math settings. A nofollow in the X-Robots-Tag header comes from the server or a plugin: remove it there, or ask your host.

The page sends Google to … Canonical link to another domain Important Clear the canonical URL in the page's Yoast or Rank Math settings, and check that the site address under Settings › General in WordPress is this domain.

The page can show up in Google No noindex in the robots meta tag or X-Robots-Tag header Important Set the page to not show in search results (noindex) and its links to not be followed (nofollow) in its Yoast or Rank Math settings.

Google follows the links on the page No nofollow in the robots meta tag or X-Robots-Tag header Notice Set the page's links to not be followed (nofollow) in its Yoast or Rank Math settings.

What each priority means: Problem priority.

How to fix it

The whole site is hidden

In WordPress, go to Settings › Reading and untick "Discourage search engines from indexing this site". This box adds noindex and nofollow to every page.

One page in Yoast SEO

Edit the page, open the Yoast SEO box and go to Advanced. "Allow search engines to show this Page in search results?" sets noindex, "Should search engines follow links on this Page?" sets nofollow. The Canonical URL field is there too.

One page in Rank Math

Edit the page, open Rank Math and go to the Advanced tab. Under Robots Meta, pick Index or No Index, and tick Nofollow for the links. The Canonical URL field is there too.

The header comes from the server

A noindex or nofollow in the X-Robots-Tag header comes from the server or a plugin, not from the settings above. Remove it there, or ask your host or developer.

For your developer

We follow redirects (up to 5) and read the final answer, and only when it answers 2xx. To see the same (the first command prints the headers of every hop; only the last one counts):

curl -sL -A 'WPMooBot/1.0 (+https://wp-moo.com/bot)' -D - -o /dev/null https://example.com/thank-you/ | grep -i x-robots-tag
curl -sL -A 'WPMooBot/1.0 (+https://wp-moo.com/bot)' https://example.com/thank-you/ \
  | grep -Eio '<meta[^>]*name=.?(robots|googlebot)[^>]*>|<link[^>]*rel=[^>]*canonical[^>]*>'

Meta and link tags are read from the first 512 KB of the decoded body. Directives for all bots and for googlebot count; header lines scoped to another bot (bingbot: noindex) don't. none counts as both noindex and nofollow. Tags inside HTML comments and scripts are ignored. A hidden page needs both:

<meta name="robots" content="noindex, nofollow">

A canonical link counts as another domain when its host differs from the page's, ignoring www; a relative one is always fine. robots.txt isn't read.

Something unclear on this page? Email [email protected]