Fix: indexing — noindex, canonical tag, robots.txt and sitemap

Indexing problems are the most expensive SEO problems: a page that Google can’t crawl or is told not to index will never rank, however good its title, content or speed. This guide covers the indexing checks in the audit report (section Technical & security) and the indexability data in a site crawl.

Find the affected pages in the app

1. Audit report: open the audit from Audits (see Read your audit score and report), go to Issues to fix and look for Indexable (Critical), Canonical tag, robots.txt, Disallow rules in robots.txt and Meta nofollow (Warnings). Page facts › Technical shows Canonical URL (Set / Not set), robots.txt (Found / Not found) and Meta robots (noindex or index, follow).
2. Keyword Score + and Web Page Audit +: the Google Lighthouse SEO section lists failed Lighthouse audits such as “Page is blocked from indexing”, “Document does not have a valid rel=canonical”, “robots.txt is not valid”, “Page has unsuccessful HTTP status code” and “Links are not crawlable”.
3. Site crawl: open the crawl from Site crawl (see Site crawl), then Overview › Indexability. It splits the crawled pages into Indexable, Noindex, Canonicalised and Not indexable (any page that did not return 200).
4. In the Pages tab, the Indexable column shows the same value per URL. Use Search URL or title to find a page, and Export to get the full list.

Indexable

How we detect it

The audit reads the page’s HTML and fails the check (Critical) when a meta name="robots" tag contains noindex. In a site crawl, pages with a noindex meta tag are counted as Noindex in Overview › Indexability. Lighthouse (+ audits) also reads the X-Robots-Tag HTTP header (“Page is blocked from indexing”, audit id is-crawlable).

Why it matters

A noindex page is removed from Google’s results. Nothing else in the report matters for that page until the noindex is gone.

Common causes

• The whole site has noindex → WordPress Discourage search engines from indexing this site is ticked (very common after a launch from staging).
• One page or post type has noindex → a Yoast SEO or Rank Math setting on that page, or for the whole content type.
• noindex in the header, not the HTML → an X-Robots-Tag: noindex header set by the server, a CDN rule or a staging plugin.
• Only on some environments → a staging copy was pushed live with its noindex settings.
• On Shopify → a theme or app adds noindex (for example to search, filtered or tag pages, which is fine) or to a page by mistake.

Fix: basic

1. Open the page in your browser, press Ctrl+U (Cmd+Option+U on Mac) and search for noindex. Note whether it is in a meta name="robots" tag.
2. WordPress, whole site: in your WordPress admin go to Settings › Reading (/wp-admin/options-reading.php), untick Discourage search engines from indexing this site and click Save Changes.
3. Yoast SEO, one page (Yoast help): edit the page, open the Yoast SEO box › Advanced and set Allow search engines to show this content in search results? to Yes (or Default for Pages, currently: Yes). Update the page.
4. Yoast SEO, whole content type: go to Yoast SEO › Settings › Content types (for example Pages or Posts) and switch Show pages in search results on. Save.
5. Rank Math, one page (Rank Math knowledge base): edit the page, open Rank Math › Advanced and set Robots Meta to Index (untick No Index). Update.
6. Rank Math, whole content type: go to Rank Math SEO › Titles & Meta › Posts (or Pages), set Robots Meta to Index and save.
7. Clear the page cache (see Platform steps) and check the source again.

Fix: expert

Check the HTTP header as well as the HTML:

curl -sI https://www.example.com/page/ | grep -i x-robots-tag

If you see X-Robots-Tag: noindex, remove it where it is set. Google’s rules for both are in Robots meta tag and X-Robots-Tag.

nginx (look in the site config and any included snippets):

# Remove or comment out lines like this for public pages:
# add_header X-Robots-Tag "noindex, nofollow" always;

# Keep noindex only where you want it, for example internal search:
location /search/ {
    add_header X-Robots-Tag "noindex, follow" always;
}

Apache (.htaccess or the virtual host):

# Remove or comment out a site-wide header like this:
# Header set X-Robots-Tag "noindex, nofollow"

# Keep it only for files you never want in Google, for example PDFs in /private/:
<If "%{REQUEST_URI} =~ m#^/private/.*\.pdf$#">
    Header set X-Robots-Tag "noindex"
</If>

The correct meta tag for a page that should rank is either no robots tag at all, or:

<meta name="robots" content="index, follow">

Platform steps

WordPress

1. Check Settings › Reading (site-wide noindex) first.
2. Check the Yoast SEO or Rank Math settings described in Fix: basic.
3. Plugins such as maintenance-mode or staging plugins can add noindex. Deactivate them on the live site.
4. Purge the cache in your caching plugin (for example LiteSpeed Cache › Toolbox › Purge All, WP Rocket › Clear cache) after the change.

Shopify

1. Shopify adds noindex automatically to some system pages (search results and similar). That is expected.
2. For a normal page, in Shopify admin go to Online Store › Themes › … › Edit code, open theme.liquid and search for noindex.
3. Remove any custom code that outputs noindex for products, collections or pages you want indexed, then Save.
4. If an SEO app adds noindex, open the app’s settings and switch the page back to indexable.

Webflow

1. Open your site from the Webflow dashboard in the Designer, open Pages, click the gear icon of the page and check SEO settings and Custom code for a noindex meta tag.
2. Check Site settings › SEO › Indexing: Disable Webflow subdomain indexing should be on for the webflow.io subdomain only. Your custom domain must not have noindex.
3. Publish the site.

Wix

1. Open your site from the Wix dashboard, open the page in the Editor, click Pages & Menu, then the … next to the page › SEO basics.
2. Turn Let search engines index this page on.
3. Check Marketing & SEO › SEO › SEO Settings for page-type rules that set noindex, then publish.

Cloudflare

1. In the Cloudflare dashboard, pick your site › Rules › Transform Rules › Modify Response Header and look for a rule that sets X-Robots-Tag.
2. Delete or narrow it, then purge the cache in Caching › Configuration › Purge Everything.

Verify the fix

1. Run curl -sI on the URL and view the source: no noindex in either place.
2. In the app, open the audit in Audits and click Run again. Indexable should pass and Page facts should show Meta robots: index, follow.
3. For a site crawl, open it in Site crawl, click Crawl again, then Compare to check that the Noindex count in Overview › Indexability dropped.
4. In Google Search Console, use URL Inspection on the URL, click Test live URL, then Request indexing.

Automation

Automatable: WordPress admin access via a plugin or the REST API (Settings › Reading, Yoast or Rank Math page meta), or a Cloudflare API token for header rules. Server headers need server SSH.

Canonical tag

How we detect it

The audit reports whether the page has a link rel="canonical" tag in its head (warning when it is missing). In a site crawl, a page whose canonical points to a different URL is counted as Canonicalised. Lighthouse (+ audits, id canonical) fails when the canonical is invalid, relative to a different domain, or points to the home page from a deep page.

Why it matters

Most pages can be reached by several URLs (with and without www, with tracking parameters, with a trailing slash or not). The canonical tells Google which URL is the main one, so ranking signals are not split between duplicates. See Google’s guide to canonical URLs.

Common causes

• No canonical at all → the theme or CMS doesn’t output one and no SEO plugin is installed.
• Every page canonicalises to the home page → a hard-coded canonical in the theme header.
• Canonical points to http:// or to the staging domain → the site URL was not updated after a migration.
• Canonical points to a URL that redirects or returns 404 → the target was moved.
• Two canonical tags on one page → the theme and an SEO plugin both output one.

Fix: basic

1. View the page source and search for rel="canonical". There should be exactly one.
2. WordPress: install or keep one SEO plugin (Yoast SEO or Rank Math). Both output a self-referencing canonical on every page automatically.
3. Yoast, custom canonical: edit the page › Yoast SEO box › Advanced › Canonical URL. Leave it empty for a self-referencing canonical; fill it only to point a duplicate at the main page.
4. Rank Math, custom canonical: edit the page › Rank Math › Advanced › Canonical URL.
5. Make sure Settings › General (/wp-admin/options-general.php) › WordPress Address (URL) and Site Address (URL) use your final https:// domain, because plugins build the canonical from them.

Fix: expert

A correct self-referencing canonical, absolute and on the final URL:

<link rel="canonical" href="https://www.example.com/services/seo-audit/">

A duplicate (for example a print or filtered version) points to the main page:

<!-- on https://www.example.com/services/seo-audit/?sort=price -->
<link rel="canonical" href="https://www.example.com/services/seo-audit/">

For files without HTML (PDFs), send the canonical as a header (Apache):

<Files "price-list.pdf">
    Header add Link "<https://www.example.com/pricing-guide/>; rel=\"canonical\""
</Files>

Remove a hard-coded canonical from a WordPress theme: search header.php (or the block theme’s head template) for rel="canonical" and delete that line when an SEO plugin is active.

Platform steps

WordPress

1. Keep one SEO plugin active. Deactivate duplicates (for example All in One SEO and Yoast together).
2. Set custom canonicals only in the Yoast or Rank Math advanced settings, as above.

Shopify

1. Shopify themes output {{ canonical_url }} in theme.liquid. Check it exists once:

<link rel="canonical" href="{{ canonical_url }}">

2. Product URLs inside collections (/collections/x/products/y) canonicalise to /products/y automatically. That is correct.

Webflow

1. In the Webflow dashboard, open the site’s Site settings › SEO › Global canonical tag URL and enter your main domain, for example https://www.example.com.
2. For one page, open its Page settings and set Canonical URL. Publish.

Wix

1. Wix adds a self-referencing canonical automatically.
2. To change one, open the site from the Wix dashboard › Editor › Pages & Menu › … › SEO basics › Advanced SEO and edit the canonical tag. Publish.

Verify the fix

1. View the source: one canonical, absolute, https://, pointing to a URL that returns 200.
2. Click Run again on the audit in Audits. Canonical tag should pass and Page facts should show Canonical URL: Set.
3. In Site crawl, open the crawl, click Crawl again and use Compare. Pages that should rank must show Yes in the Indexable column, not Canonicalised.
4. In Search Console URL Inspection, check User-declared canonical and Google-selected canonical match.

Automation

Automatable on WordPress with admin access via the SEO plugin’s REST fields; assisted on other platforms (theme edits need Shopify theme access or Webflow Designer access).

robots.txt

How we detect it

The audit requests /robots.txt on the page’s domain and warns when it is not found. Page facts › Technical shows robots.txt: Found or Not found. Lighthouse (+ audits, id robots-txt) fails when the file returns an error or has invalid lines.

Why it matters

robots.txt tells search engines what they may crawl and where your sitemap is (Google’s robots.txt guide). A missing file is not a disaster (everything is allowed), but a broken one, or one that answers with a 5xx error, can make Google stop crawling the whole site.

Common causes

• The file does not exist → no CMS rule or plugin creates it.
• It returns 403 or 5xx → a firewall or server rule blocks it.
• It returns an HTML page → the CMS serves a 404 page with status 200.
• It has typos → for example Disalow, missing colons, or rules before any User-agent line.

Fix: basic

1. Open https://yourdomain.com/robots.txt in a browser. It should show plain text.
2. WordPress with Yoast: in your WordPress admin go to Yoast SEO › Tools › File editor and click Create robots.txt file (or edit it). Save.
3. WordPress with Rank Math: go to Rank Math SEO › General Settings › Edit robots.txt, enter the rules and save.
4. Shopify: robots.txt exists automatically. To change it, see Platform steps › Shopify.
5. Webflow: in the Webflow dashboard, open the site’s Site settings › SEO › robots.txt, paste the rules and publish.
6. Wix: in the Wix dashboard, go to Marketing & SEO › SEO › Robots.txt Editor (under Tools and settings), edit and save.

Fix: expert

A safe default robots.txt for most sites:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.example.com/sitemap_index.xml

For a non-WordPress site, drop the two wp-admin lines. Upload the file as robots.txt to the web root (the same folder as the home page index file) with UTF-8 encoding.

If a firewall blocks it, allow it explicitly (nginx):

location = /robots.txt {
    allow all;
    log_not_found off;
    access_log off;
    try_files $uri /index.php?$args;
}

Platform steps

Shopify

1. In Shopify admin, go to Online Store › Themes › … › Edit code.
2. Click Add a new template, choose robots.txt and create it. This creates robots.txt.liquid.
3. Keep Shopify’s default rules and add your own lines, for example:

{% for group in robots.default_groups %}
  {{- group.user_agent }}
  {%- for rule in group.rules %}
    {{ rule }}
  {%- endfor %}
  {%- if group.user_agent.value == '*' %}
    {{ 'Disallow: /*?q=*' }}
  {%- endif %}
  {%- if group.sitemap != blank %}
    {{ group.sitemap }}
  {%- endif %}
{% endfor %}

4. Save and open /robots.txt to check the result.

Cloudflare

1. If robots.txt shows a challenge page, add a WAF custom rule that skips security for http.request.uri.path eq "/robots.txt" (in the Cloudflare dashboard, pick your site › Security › WAF › Custom rules › Create rule, action Skip; see WAF custom rules). Full steps: Fix: Cloudflare blocking the crawler.

Verify the fix

1. Open /robots.txt: plain text, status 200 (check with curl -I https://www.example.com/robots.txt).
2. Click Run again on the audit in Audits. robots.txt should pass and Page facts should show Found.
3. In Search Console, open Settings › robots.txt to see the version Google fetched and any errors.

Automation

Automatable with WordPress admin access via the SEO plugin, Webflow or Wix site settings access, or Shopify theme access (Admin API with theme write scope).

Disallow rules in robots.txt

How we detect it

The scanner reads robots.txt and flags Disallow rules in robots.txt when it finds rules that may block search engines from pages. The site crawl also obeys robots.txt: blocked URLs are skipped and don’t appear in the crawl at all. See Fix: Crawl finds few pages.

Why it matters

One wrong line (Disallow: /) stops Google from crawling the entire site. Overly broad rules can also block CSS and JavaScript, so Google can’t render the page.

Common causes

• Disallow: / left over from a staging site → the site was launched with its development robots.txt.
• Blocking /wp-content/ or /assets/ → CSS, JS and images can’t be fetched, so Google sees a broken page.
• Wildcards that match too much → Disallow: /*? also blocks paginated or important parameter URLs.
• A plugin rewrites robots.txt → a security or SEO plugin adds its own rules.

Fix: basic

1. Open /robots.txt and read every Disallow line under User-agent: * and User-agent: Googlebot.
2. Delete Disallow: / unless the site really must be hidden.
3. Delete rules that block theme, plugin or asset folders (/wp-content/, /wp-includes/, /assets/, /static/).
4. Keep rules only for private or duplicate areas: admin, cart, checkout, account, internal search.
5. Save the file using the editor for your platform (see robots.txt › Fix: basic above).

Fix: expert

Before and after for a typical mistake:

# Wrong: blocks the whole site
User-agent: *
Disallow: /

# Right: blocks only private areas
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /?s=
Sitemap: https://www.example.com/sitemap.xml

Rules are prefix matches, * matches any characters and $ anchors the end. The most specific (longest) matching rule wins, and Allow beats Disallow on a tie.

Platform steps

WordPress

1. If Settings › Reading › Discourage search engines is ticked, WordPress serves Disallow: / from its virtual robots.txt. Untick it.
2. If a physical robots.txt file exists in the web root, it overrides the virtual one. Edit it with the Yoast or Rank Math file editor (see robots.txt › Fix: basic), or in your hosting control panel’s cPanel › File Manager › public_html.

Shopify

1. Shopify’s default rules are safe. Only edit robots.txt.liquid as shown above, and never add Disallow: /.

Verify the fix

1. Open /robots.txt and confirm the new rules are live (purge the CDN cache if not).
2. In Search Console, open Settings › robots.txt and click Request a recrawl of the file.
3. Click Run again on the audit in Audits, and Crawl again on the crawl in Site crawl. Previously skipped pages should now appear in the Pages tab.

Automation

Assisted: the edit is simple and automatable with CMS or file access, but a person should confirm which areas must stay private.

Meta nofollow

How we detect it

The scanner flags Meta nofollow when the page has meta name="robots" with nofollow. This tells search engines not to follow any link on the page.

Why it matters

Links pass ranking value and help Google discover pages. A page-level nofollow cuts every internal link on the page, which can leave other pages undiscovered.

Common causes

• The SEO plugin’s robots meta for the page or content type is set to nofollow.
• A staging or maintenance plugin adds noindex, nofollow to every page.
• A theme template hard-codes nofollow in the head.

Fix: basic

1. View the source and search for nofollow inside the robots meta tag.
2. Yoast SEO: edit the page › Yoast SEO › Advanced › Should search engines follow links on this content? › Yes. Update.
3. Rank Math: edit the page › Rank Math › Advanced › untick No Follow under Robots Meta. Update.
4. Purge the page cache.

Fix: expert

Replace the tag in the template, or remove it entirely (follow is the default):

<!-- Wrong -->
<meta name="robots" content="index, nofollow">
<!-- Right -->
<meta name="robots" content="index, follow">

Use rel="nofollow", rel="sponsored" or rel="ugc" on single links instead, for paid or user-generated links.

Platform steps

Shopify

1. In Shopify admin › Online Store › Themes › … › Edit code, search theme.liquid and snippets for nofollow in a robots meta tag and remove it.

Webflow

1. In the Webflow dashboard, check the page’s Custom code › Inside head tag and Site settings › Custom code for a robots meta with nofollow. Remove it and publish.

Verify the fix

1. View the source: no nofollow in the robots meta.
2. Click Run again on the audit in Audits. Meta nofollow should pass.

Automation

Automatable with WordPress admin access via the SEO plugin, or theme access on Shopify and Webflow.

XML sitemap

How we detect it

The app does not grade the sitemap as a separate check. The audit shows whether robots.txt is found (where the sitemap should be listed), and the site crawl shows which pages are indexable. Use both with the steps below.

Why it matters

A sitemap lists the URLs you want indexed, so Google finds new and deep pages faster and you can monitor indexing per URL in Search Console.

Common causes

• No sitemap → no SEO plugin, or the platform feature is off.
• Sitemap lists bad URLs → noindex, redirected, canonicalised or 404 pages are included.
• Sitemap not linked → it is not in robots.txt and was never submitted.

Fix: basic

1. Yoast SEO: go to Yoast SEO › Settings › Site features and switch XML sitemaps on. The sitemap is at /sitemap_index.xml.
2. Rank Math: go to Rank Math SEO › Sitemap Settings, enable the module and choose which post types are included. The sitemap is at /sitemap_index.xml.
3. Shopify: the sitemap is automatic at /sitemap.xml.
4. Webflow: in the Webflow dashboard, open the site’s Site settings › SEO › Sitemap and switch Auto-generate sitemap on.
5. Wix: the sitemap is automatic at /sitemap.xml once the site is published and connected to a domain.
6. Add the sitemap line to robots.txt: Sitemap: https://www.example.com/sitemap_index.xml.
7. In Google Search Console, open Indexing › Sitemaps, enter the sitemap URL and click Submit. No property yet? Add and verify your site first.

Fix: expert

A minimal valid sitemap if you must build one by hand:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example.com/</loc>
    <lastmod>2026-10-20</lastmod>
  </url>
  <url>
    <loc>https://www.example.com/services/</loc>
    <lastmod>2026-10-01</lastmod>
  </url>
</urlset>

Only list final, indexable, 200-status, canonical URLs. Keep lastmod honest (the date the content really changed). Google ignores priority and changefreq.

Platform steps

WordPress

1. Exclude thin archives: in Yoast SEO › Settings › Content types / Categories & tags, switch Show … in search results off for types you don’t want; they leave the sitemap too.

Verify the fix

1. Open the sitemap URL: valid XML, status 200.
2. In Search Console Indexing › Sitemaps, the status should be Success. Then open Indexing › Pages to see which URLs are indexed and why others are not.
3. Compare the sitemap URLs with the Pages tab of a fresh crawl in Site crawl: every sitemap URL should be crawlable and Indexable: Yes.

Automation

Automatable with WordPress admin access via the SEO plugin, plus Search Console access to submit (the Search Console API supports sitemap submission).

Lighthouse SEO audits (+ audits)

How we detect it

Keyword Score + and Web Page Audit + run Google Lighthouse. Failed SEO audits appear in the Google Lighthouse section with their Lighthouse title. The common indexing ones: “Page is blocked from indexing” (is-crawlable), “Document does not have a valid rel=canonical” (canonical), “robots.txt is not valid” (robots-txt), “Page has unsuccessful HTTP status code” (http-status-code), “Links are not crawlable” (crawlable-anchors). Exact titles vary slightly between Lighthouse versions.

Why it matters

These are the same blockers Google’s own crawler meets. Each one can stop a page from being indexed or stop Google from following links.

Common causes

• is-crawlable → a noindex meta tag or X-Robots-Tag header (see Indexable).
• canonical → a canonical to another domain, a relative or broken URL, or several conflicting canonicals.
• robots-txt → robots.txt returns 5xx or contains invalid lines.
• http-status-code → the page returns 4xx or 5xx to the audit browser.
• crawlable-anchors → links built as a tags without a real href, such as href="javascript:void(0)" or click handlers only.

Fix: basic

1. For is-crawlable, canonical and robots-txt, follow the sections above: Indexable, Canonical tag, robots.txt.
2. For http-status-code, open the page and check it loads; if it’s a block, see Fix: Site can’t be crawled.
3. For crawlable-anchors, ask your page builder to use normal link elements (most builders have a Link setting with a URL field: use it instead of an on-click action).

Fix: expert

Make every navigational element a real link:

<!-- Not crawlable -->
<a onclick="goTo('/services/')">Services</a>
<span data-href="/contact/">Contact</span>

<!-- Crawlable -->
<a href="/services/">Services</a>
<a href="/contact/">Contact</a>

In React or Vue routers, render Link / router-link components, which output an a href.

Platform steps

WordPress

1. In Elementor, select the button or icon › Content › Link and enter the URL, rather than using a popup or JavaScript action for navigation.

Verify the fix

1. Click Run again on the + audit in Audits. The audits should move out of the failed list.
2. Optional: run Lighthouse yourself in Chrome DevTools (Lighthouse tab › SEO), or test the URL in PageSpeed Insights.

Automation

Assisted: detection is automatic; fixes need CMS or code access depending on the cause.

Common questions

Related articles

• Fix: URLs with parameters and URL structure
• Fix: HTTPS, SSL and redirects (redirect chains, mixed content)
• Fix: Crawl finds few pages — redirects, robots.txt, login walls and JavaScript links
• Crawl a whole website
• Read your audit score and report

Still need help?

Ask in the chat inside the app, or send us a message on the contact page. You can also browse all help topics.