Fix: Crawl finds few pages — redirects, robots.txt, login walls and JavaScript links

When a crawl ends with far fewer pages than the site has, the crawler either couldn’t load the start page, couldn’t see the links, or wasn’t allowed to follow them. This guide covers every cause except blocks and errors, which are in Fix: Site can’t be crawled.

How the crawler finds pages

• It starts at the client’s website URL (the home page).
• It reads the HTML and follows every <a href> link on the same site. www and non-www count as the same site. Subdomains count only with Include subdomains.
• It skips links that start with #, mailto:, tel: or javascript:, links to files (images, PDFs, CSS, JS, ZIP, video and similar), and URLs blocked by robots.txt.
• It doesn’t read the XML sitemap and doesn’t submit forms.
• Render JavaScript renders only the start page, so links that JavaScript adds on deeper pages aren’t found.
• It stops at Max pages (500 or 1,000) or when its time budget runs out (about 30 minutes of crawling).

Find the cause in the app

1. Open Site crawl and click the crawl.
2. On Pages, look at the first row (the home page) and its Status.
3. Status 301/302 → see “Redirects to another host” or “Redirect loops”. No response with a redirecting URL → “Redirect loops”.
4. Status 401, or a 200 with very few words → “Login wall”.
5. Status 200 and only 1–3 pages → “JavaScript-rendered links”.
6. Pages you expected are missing and https://example.com/robots.txt has Disallow rules for them → “robots.txt blocking”. Blocked URLs are skipped silently, so they don’t appear in the Pages tab at all.
7. Crawled pages equals your Max pages → “Max pages and time budget”.

Redirects to another host

How we detect it

The home page redirects to a different domain (for example example.com → shop.example.net, or → example.de). The crawler follows the redirect, but links on the final site are on another host, so they count as External and aren’t crawled. The crawl ends with 1 page or very few.

Why it matters

The client’s website in the app is not where the content lives, so the crawl measures almost nothing.

Common causes

• If the redirect always goes to the same other domain → the site moved, or the shop is on a separate domain.
• If the redirect goes to a country or language version → geo-redirection based on our server’s location (IP-based country detection, Shopify Markets, a geolocation plugin).
• If it redirects to a subdomain (www.example.com → blog.example.com) → the content is on a subdomain.

Fix: basic

1. Find the final URL:

curl -sS -o /dev/null -L -w "%{num_redirects} redirects, final: %{url_effective}\n" https://example.com/

2. If the site moved for good, open Clients, open the client › Settings tab, and set the website to the final domain. See Edit or delete a client.
3. If the content is on a subdomain, start a new crawl in Site crawl with Include subdomains on.
4. If it’s a geo-redirect, turn off automatic redirection for bots, or for our IPs (platform steps below).
5. Click Crawl again.

Fix: expert

Geo-redirects should never be forced on crawlers: Googlebot mostly crawls from the US and would be redirected too. Show a banner suggesting the local version instead of redirecting. If you must redirect, exclude our IPs (and known crawlers):

geo $no_geo_redirect {
    default                 0;
    147.93.19.85            1;
    2a02:4780:12:d33e::1    1;
}
# in your geo-redirect logic:
# if ($no_geo_redirect = 0) { return 302 https://example.de$request_uri; }

Platform steps

Shopify

In Shopify admin › Settings › Markets › Preferences (or the market’s Domains and languages): turn off automatic redirection, or remove the geolocation app’s auto-redirect and use its banner mode instead.

WordPress

Geolocation plugins (for example WPML’s browser-language redirect at WPML › Languages › Browser language redirect) should be set to Disable browser language redirect.

Verify the fix

The curl command ends on the client’s website domain. Crawl again finds the site’s pages.

Automation

Assisted: needs app access to update the client website, plus platform admin for geo-redirect settings.

Redirect loops

How we detect it

A URL redirects more than 5 times. The crawler gives up and the page shows No response. Browsers show “redirected you too many times”.

Why it matters

A loop makes the page unreachable for everyone, not only for us.

Common causes

• If Cloudflare is in front with SSL/TLS mode Flexible and the server forces https → Cloudflare asks over http, the server redirects to https, forever.
• If WordPress loops between www and non-www → WordPress Address and Site Address don’t match the server’s redirect.
• If two plugins redirect (for example an SSL plugin and a redirect plugin) → they undo each other.
• If a cookie or language redirect bounces visitors without cookies → bots (no cookies) loop.

Fix: basic

1. Trace the hops:

curl -sS -I -L --max-redirs 10 https://example.com/ 2>&1 | grep -iE "^HTTP|^location"

2. If the location alternates between http and https on a Cloudflare site → in the Cloudflare dashboard, pick your site › SSL/TLS › Overview and set it to Full (strict) (SSL/TLS modes).
3. If it alternates between www and non-www → in WordPress admin (/wp-admin/options-general.php) open Settings › General and set both WordPress Address (URL) and Site Address (URL) to the one version you want, starting with https://.
4. If a plugin is involved → deactivate SSL or redirect plugins one at a time and test again.
5. Click Crawl again.

Fix: expert

Use one canonical redirect in one place. Apache .htaccess (https + non-www in one hop):

RewriteEngine On
RewriteCond %{HTTPS} off [OR]
RewriteCond %{HTTP_HOST} ^www\.example\.com$ [NC]
RewriteRule ^(.*)$ https://example.com/$1 [R=301,L]

Behind Cloudflare or another proxy, %{HTTPS} is always off at the origin with Flexible SSL. Use Full (strict), or check the forwarded header:

RewriteCond %{HTTP:X-Forwarded-Proto} !https
RewriteRule ^(.*)$ https://%{HTTP_HOST}/$1 [R=301,L]

nginx:

server {
    listen 80;
    listen [::]:80;
    server_name example.com www.example.com;
    return 301 https://example.com$request_uri;
}
server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name www.example.com;
    return 301 https://example.com$request_uri;
}

If WordPress admin is unreachable, set the URLs in wp-config.php:

define( 'WP_HOME', 'https://example.com' );
define( 'WP_SITEURL', 'https://example.com' );

Platform steps

Cloudflare

In the Cloudflare dashboard › your site › SSL/TLS › Overview › Full (strict), and SSL/TLS › Edge Certificates › Always Use HTTPS on. Remove duplicate http→https Page Rules or Redirect Rules.

WordPress

With WP-CLI: wp option get home and wp option get siteurl must match.

Verify the fix

The trace shows at most one redirect, then HTTP/2 200. Crawl again shows real status codes. More on redirects: Fix: HTTPS, SSL and redirects.

Automation

Automatable: needs a Cloudflare API token (SSL mode) or WordPress admin / file access.

robots.txt blocking

How we detect it

The crawler reads /robots.txt first and skips URLs disallowed for SEOAuditSoftware, or for * when there is no group for us. If robots.txt doesn’t answer with status 200, everything is allowed. Blocked URLs are skipped silently: they don’t appear in the Pages tab or in Overview › Indexability, so compare the missing sections with the Disallow lines in robots.txt.

Why it matters

Rules that block us usually block Google too. A leftover Disallow: / from development keeps the whole site out of search.

Common causes

• If robots.txt has User-agent: * then Disallow: / → everything is blocked for every bot.
• If a group lists many SEO tools by name → we may be included, or a broad pattern matches.
• If whole folders are disallowed (for example /shop/ or /blog/) → those sections aren’t crawled.

Fix: basic

1. Open https://example.com/robots.txt (your own domain). Background: Google’s robots.txt guide.
2. If the site should be public, remove Disallow: / under User-agent: *.
3. To let only our crawler in, add the group below at the top of the file.
4. Save, then click Crawl again in Site crawl.

Fix: expert

Allow our crawler everywhere:

User-agent: SEOAuditSoftware
Allow: /

A safe default robots.txt for most WordPress sites:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap_index.xml

A crawler follows only the most specific group that matches its name. If you add a SEOAuditSoftware group, the * rules no longer apply to us, so add any folders you still want us to skip to our group too.

Platform steps

WordPress (Yoast SEO)

In WordPress admin, Yoast SEO › Tools › File editor › Create robots.txt file (or edit the existing one), then Save changes to robots.txt. Help: Yoast help.

WordPress (Rank Math)

Rank Math SEO › General Settings › Edit robots.txt, then Save Changes. Help: Rank Math knowledge base.

WordPress setting

Settings › Reading › Search engine visibility: “Discourage search engines from indexing this site” adds a noindex and, on some versions, changes the virtual robots.txt. Untick it on live sites.

Shopify

In Shopify admin › Online Store › Themes › … › Edit code, add a robots.txt.liquid template in Templates, and add the group for us. Don’t remove Shopify’s default rules.

Wix

In the Wix dashboard › Settings › SEO (or Marketing & SEO › SEO) › Robots.txt editor › View File, edit, Save.

Verify the fix

https://example.com/robots.txt shows the new rules. In Google, check it in Search Console › Settings › robots.txt report. Crawl again: the previously missing sections now appear in the Pages tab.

Automation

Automatable: needs WordPress admin (Yoast or Rank Math), Shopify theme access, or file access.

Login wall

How we detect it

The home page returns 401 (basic authentication) or redirects to a login page, or shows a password form with very little text. The crawler can’t sign in, so it finds 1 page.

Why it matters

Search engines can’t sign in either. Only public pages can rank.

Common causes

• If a browser pop-up asks for a username and password → HTTP basic authentication (staging sites).
• If you see the store’s password page → platform password protection (Shopify, Squarespace, Wix, Webflow).
• If you see a launch or maintenance page → a maintenance plugin.
• If member content is behind a login → membership plugins; only the public pages can be crawled.

Fix: basic

1. Crawl the public live site, not the staging copy.
2. If the live site is still password protected, remove the protection when the client is ready to launch (platform steps below).
3. Turn off maintenance or launch-page plugins on live sites.
4. Click Crawl again.

Fix: expert

Let our IPs into a protected staging site without a password.

Apache 2.4 (.htaccess):

AuthType Basic
AuthName "Staging"
AuthUserFile /home/user/.htpasswd
<RequireAny>
    Require ip 147.93.19.85
    Require ip 2a02:4780:12:d33e::1
    Require valid-user
</RequireAny>

nginx:

location / {
    satisfy any;
    allow 147.93.19.85;
    allow 2a02:4780:12:d33e::1;
    deny all;
    auth_basic "Staging";
    auth_basic_user_file /etc/nginx/.htpasswd;
    try_files $uri $uri/ /index.php?$args;
}

Remember: staging sites usually also send noindex, so the crawl will report them as Noindex.

Platform steps

Shopify

In Shopify admin › Online Store › Preferences › Password protection: untick Restrict access to visitors with the password, click Save. Help: Shopify online store preferences.

Squarespace

In your Squarespace account, open the site › Settings › Site availability (or Site visibility): choose Public.

Wix

Publish the site, and in the Wix dashboard › Settings › Members (or the page’s Permissions) make sure pages aren’t set to Members only or Password holders.

Webflow

Site settings › General › Password protection (or Publishing): turn off site-wide password protection; page-level passwords are in each page’s settings.

Verify the fix

curl -sS -o /dev/null -w "%{http_code}\n" https://example.com/ returns 200 without credentials. Crawl again.

Automation

Manual: the site owner decides when to make the site public; then platform admin, WordPress admin or file access.

JavaScript-rendered links

How we detect it

The home page returns 200 but the crawl finds only 1–3 pages. View the page source (Ctrl+U): the menu links aren’t there as <a href> links.

Why it matters

Google renders JavaScript later and less reliably than it reads HTML (Google: JavaScript SEO basics). Many AI crawlers don’t run JavaScript at all. Plain links get pages found faster by every crawler.

Common causes

• If the site is a single-page app (React, Vue, Angular) without server rendering → the HTML is an empty shell.
• If the menu is built by a script or loaded with an API call → no links in the HTML.
• If navigation uses <button> or <div onclick> instead of <a href> → there are no links to follow, even after rendering.
• If links use href="javascript:void(0)" or href="#" with a click handler → skipped.

Fix: basic

1. Start a new crawl in Site crawl with Render JavaScript on. It renders the start page in headless Chrome, so links added by JavaScript on the home page are found.
2. Add a plain HTML footer with links to the main sections.
3. Add an HTML sitemap page (a normal page listing links to all important pages) and link it in the footer.
4. Click Crawl again.

Fix: expert

Serve real links in the HTML:

<!-- Crawlable -->
<a href="/services/seo-audit/">SEO audit</a>

<!-- Not crawlable -->
<span onclick="location.href='/services/seo-audit/'">SEO audit</span>
<a href="javascript:void(0)" onclick="go('/services/seo-audit/')">SEO audit</a>

Render content on the server:

• Next.js: use server components or getStaticProps/getServerSideProps and the Link component (it outputs <a href>).
• Nuxt: keep ssr: true (the default) and use NuxtLink.
• React (Vite/CRA) without a framework: pre-render routes at build time (for example with a static-site generation step) or move to a framework with SSR.
• Angular: use Angular SSR (ng add @angular/ssr).

Check what non-JavaScript crawlers see:

curl -sS -A "SEOAuditSoftware" https://example.com/ | grep -oE '<a [^>]*href="[^"#]+' | head -n 20

Platform steps

WordPress

Page builders output normal links for menus built in Appearance › Menus or the theme’s header builder. Avoid “mega menu” widgets that load links with AJAX.

Wix, Squarespace, Shopify, Webflow

They render links on the server, so this cause is rare. Custom code embeds that build menus with JavaScript are the exception.

Verify the fix

The curl command lists the menu links. Crawl again finds many more pages.

Automation

Manual: needs code or theme changes on the site.

Max pages and time budget

How we detect it

Crawled pages equals the Max pages you chose (500 or 1,000), or the crawl stopped earlier on a slow site because its time budget (about 30 minutes of crawling) ran out.

Why it matters

Pages beyond the limit aren’t checked, so issues there are missing from the report.

Common causes

• If the site has more pages than Max pages → the limit was reached.
• If pages take several seconds each → fewer pages fit in the time budget.
• If the site has endless URL variations (filters, calendars, session IDs, ?sort=) → the crawler spends the budget on near-duplicates.

Fix: basic

1. Start the crawl in Site crawl with Max pages 1,000.
2. Speed up the site (caching, CDN): see Fix: page speed.
3. Cut endless URL variations: see Fix: URLs with parameters.
4. Click Crawl again.

Fix: expert

Block crawl traps for all bots in robots.txt (only for URLs that should never rank):

User-agent: *
Disallow: /*?*sort=
Disallow: /*?*filter_
Disallow: /calendar/

Platform steps

Shopify

Collection filter URLs (?filter.p.) and sort URLs multiply quickly; Shopify’s default robots.txt already blocks some of them.

Verify the fix

Crawled pages stays under the limit and the pages you care about appear in the Pages tab.

Automation

Assisted: needs robots.txt access and performance work on the site.

Pages only in the sitemap

How we detect it

Pages you know exist (listed in the XML sitemap) are missing from the Pages tab, while the crawl finished normally.

Why it matters

If no page links to a page, visitors can’t reach it and Google treats it as less important.

Common causes

• If the page is only reachable from the sitemap or from ads → nothing links to it.
• If links to it are inside a form, a search result or a JavaScript widget → the crawler can’t follow them.

Fix: basic

1. Add a link to each missing page from a relevant page, the menu or the footer. How: Fix: orphan pages.
2. Click Crawl again.

Fix: expert

Compare the sitemap with the crawl export: in Site crawl, open the crawl and download it with Export, list the sitemap URLs, and find URLs in the sitemap that aren’t in the crawl:

curl -sS https://example.com/sitemap.xml | grep -oE "<loc>[^<]+" | sed 's/<loc>//' | sort > sitemap.txt
# crawl.txt = the URL column from the crawl export, one URL per line
sort crawl.txt > crawl-sorted.txt
comm -23 sitemap.txt crawl-sorted.txt

(For a sitemap index, run the first command on each child sitemap.) More in Fix: orphan pages and internal links.

Platform steps

WordPress

Use a related-posts block or the theme’s breadcrumbs so every post links to its category and to similar posts.

Verify the fix

Crawl again: the missing pages now appear in the Pages tab.

Automation

Assisted: needs CMS access to add internal links.

Common questions

Related articles

• Fix: Site can't be crawled (server errors, blocks and timeouts)
• Fix: Cloudflare blocks the crawl or audit (403, 'Just a moment', 429)
• Fix: HTTPS, SSL and redirects (redirect chains, mixed content)
• Fix: orphan pages and internal links
• Fix: Crawling Shopify, Wix, Squarespace and Webflow sites
• Crawl a whole website

Still need help?

Ask in the chat inside the app, or send us a message on the contact page. You can also browse all help topics.