Fix: Site can't be crawled (server errors, blocks and timeouts)

When a crawl finds only one page, shows No response, or an audit stops with a 403 or a timeout, the client’s website did not give our server a normal page. This guide shows how to recognise each cause and fix it, step by step. Every section ends with a link to a deeper guide.

How a blocked site shows in the app

What to allowlist

What to allowlist for SEO Audit Software

• IPv4: 147.93.19.85
• IPv6: 2a02:4780:12:d33e::1
• User-Agent (Site crawl, GEO and Local SEO audits): Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0 Safari/537.36 SEOAuditSoftware (match on the token SEOAuditSoftware)
• robots.txt user agent token: SEOAuditSoftware
• Classic audits (Keyword Score, Web Page Audit, Keyword Score + and Web Page Audit +) and the crawl option Render JavaScript load the page in headless Chrome with a normal browser User-Agent that has no SEOAuditSoftware token. Allowlist these by IP.

All crawls and audits leave from our own server through its own outbound proxy, so these are the addresses the site sees. If they ever change, ask in the in-app chat for the current list.

Allowlist by IP whenever you can. A User-Agent can be copied by anyone, and the classic audits don’t send our token. A User-Agent rule is fine as an extra for crawls.

How the crawler behaves

Knowing this helps you set firewall and rate limits correctly.

• One request at a time, with a 0.25 second pause between pages. That is a few requests per second at most, usually fewer.
• Each page gets 10 seconds to connect and 25 seconds to send data.
• Up to 5 redirects per URL. More than 5 counts as no response.
• Only http and https on the normal ports 80 and 443.
• It reads up to 3 MB of HTML per page.
• It reads robots.txt first and obeys the SEOAuditSoftware group, or the * group if there is none. If robots.txt doesn’t answer with status 200, everything is allowed.
• It stays on the same site. www and non-www count as the same site. Subdomains are crawled only with Include subdomains.
• It follows links found in the HTML. It does not read the XML sitemap.
• After the pages, it sends HEAD requests to up to 150 images, CSS and JavaScript files to check their status and size.
• Render JavaScript renders only the start page in headless Chrome, to find links that appear after JavaScript runs.

Quick triage: test from the command line

Run these from any computer with a terminal (macOS, Linux, or Windows with Git Bash or WSL). Replace example.com with the client’s domain.

1. Request the home page with our User-Agent and look at the status line and the server header:

curl -sS -I -A "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0 Safari/537.36 SEOAuditSoftware" https://example.com/

2. Measure the status code and the total time:

curl -sS -o /dev/null -A "SEOAuditSoftware" -w "%{http_code} %{time_connect}s connect, %{time_starttransfer}s first byte, %{time_total}s total\n" https://example.com/

3. Follow redirects and list every hop (more than 5 hops fails):

curl -sS -o /dev/null -L --max-redirs 10 -w "%{num_redirects} redirects, final URL: %{url_effective}\n" https://example.com/

4. Check DNS for IPv4 and IPv6 (a wrong AAAA record breaks our IPv6 requests):

dig +short A example.com
dig +short AAAA example.com
dig +short A www.example.com

On Windows without dig, use nslookup example.com and nslookup -type=AAAA example.com.

5. Check the SSL certificate and chain:

openssl s_client -connect example.com:443 -servername example.com -showcerts </dev/null 2>/dev/null | grep -E "subject=|issuer=|Verify return code"

Verify return code: 0 (ok) is good. Anything else is an SSL problem.

6. Look at the body of an error page to see who blocked you:

curl -sS -A "SEOAuditSoftware" https://example.com/ | head -c 1500

7. Read the result with the table in the next section.

Important: curl from your own computer uses your IP, not ours. A 200 here plus a block in the app means the block is IP-based or reputation-based (data-centre IPs). That points to a firewall, CDN or security plugin.

Symptom → cause → fix

Timeout

How we detect it

The page did not connect within 10 seconds or sent no data for 25 seconds. The Pages tab shows No response and the page is counted in Server errors (5xx). Audits show The audit took too long or The page could not be downloaded (timeout, DNS error or blocked address).

Why it matters

If our server waits too long, Googlebot does too. Google slows down crawling of sites that answer slowly, so new pages get indexed later.

Common causes

• If every page times out → the firewall silently drops our IP (no error, just silence), or the server is overloaded.
• If only some pages time out → those pages are slow to build (uncached, heavy database queries, search or filter pages).
• If the home page loads fast in your browser but not for us → your browser gets a cached copy; we hit an uncached page, or our IP is dropped.

Fix: basic

1. Open the page in a private browser window and time it, or test it in PageSpeed Insights. If it takes more than 10 seconds, the site is slow for everyone.
2. Turn on page caching (WordPress: LiteSpeed Cache, WP Rocket or W3 Total Cache; most hosts have a cache switch in the panel).
3. Allowlist 147.93.19.85 and 2a02:4780:12:d33e::1 in the firewall or security plugin (see What to allowlist).
4. In Site crawl, open the crawl and click Crawl again (or Run again on the audit in Audits).

Fix: expert

Find out whether the server is slow or the connection is dropped:

curl -sS -o /dev/null -w "connect %{time_connect}s, first byte %{time_starttransfer}s, total %{time_total}s\n" --max-time 40 https://example.com/

A long connect time points to a firewall drop or network issue. A long first byte time points to slow PHP, database or upstream. Raise server timeouts only after fixing the slowness: see Fix: No response › Timeout.

Platform steps

WordPress

Install a cache plugin, and use Query Monitor to find slow plugins and queries on the page that times out.

Cloudflare

In the Cloudflare dashboard, pick the site and check Security › Events for our IP (how to read Security Events). Cloudflare returns its own 524 error if the origin takes more than 100 seconds.

nginx

Look in /var/log/nginx/error.log for upstream timed out lines.

Verify the fix

Run the curl timing command again: total time well under 10 seconds. Then click Crawl again and check the Pages tab: no No response rows. Use Compare to see Server errors (5xx) under fixed issues.

Automation

Assisted: needs server SSH or hosting panel access to read logs and change caching or firewall settings.

DNS error

How we detect it

Our server could not turn the domain name into an IP address, or got an address that doesn’t answer. The page shows No response.

Why it matters

If the name doesn’t resolve for us, it may not resolve for some visitors or for Googlebot either. A broken AAAA (IPv6) record often fails silently for some networks only.

Common causes

• If dig +short A example.com is empty → the A record is missing or the domain expired.
• If dig +short AAAA example.com returns an address the server doesn’t listen on → a stale IPv6 record. Our server has IPv6, so we try it; many home connections don’t, so the site “works for everyone”.
• If only www or only non-www fails → one of the two records is missing.
• If DNS was changed in the last 48 hours → old records may still be cached.

Fix: basic

1. Log in to the DNS provider (registrar, the Cloudflare dashboard, or the host’s DNS zone editor). To see what the world sees first, look the domain up in Google Admin Toolbox Dig.
2. Check there is an A record for @ and for www (or a CNAME for www) pointing to the host’s IP.
3. Delete any AAAA record that doesn’t point to the current server, or set it to the server’s correct IPv6 address.
4. Wait for the TTL to pass (often 5 minutes to 1 hour), then click Crawl again.

Fix: expert

dig +short A example.com
dig +short AAAA example.com
curl -6 -sS -I https://example.com/
curl -4 -sS -I https://example.com/

If curl -4 works and curl -6 fails, fix or remove the AAAA record. Full steps for every DNS case are in Fix: No response › DNS error.

Platform steps

Cloudflare

In the Cloudflare dashboard › your site › DNS › Records: check A, AAAA and CNAME records for @ and www. Proxied (orange cloud) records hide the origin IP; that is fine.

cPanel

Domains › Zone Editor › Manage: check the A and AAAA records.

Verify the fix

dig +short A and dig +short AAAA return only addresses that answer, and both curl -4 and curl -6 return 200. Then Crawl again.

Automation

Assisted: needs DNS provider access (for example a Cloudflare API token with DNS edit rights).

SSL/TLS error

How we detect it

The secure connection failed before any page was sent. The Pages tab shows No response. We check certificates like a browser does, so an invalid certificate stops the crawl.

Why it matters

Browsers show a full-page security warning for the same problem, and Google won’t index pages it can’t load securely.

Common causes

• If the certificate date has passed → it expired (auto-renewal failed).
• If browsers work but openssl says unable to get local issuer certificate → the server doesn’t send the intermediate certificate (desktop browsers often fill the gap; servers don’t).
• If the error says the name doesn’t match → the certificate covers example.com but not www.example.com, or the reverse.
• If the server only supports TLS 1.0/1.1 → too old; TLS 1.2 or 1.3 is required.

Fix: basic

1. Open the site in a browser and click the padlock to see the expiry date and the names covered, or run the SSL Labs test for a full report (chain, names, TLS versions).
2. Renew or reissue the certificate in the hosting panel (cPanel: Security › SSL/TLS Status › Run AutoSSL; Plesk: Websites & Domains › SSL/TLS Certificates › Install free Let’s Encrypt; Hostinger hPanel: Security › SSL).
3. Make sure both example.com and www.example.com are included.
4. Click Crawl again.

Fix: expert

openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null | openssl x509 -noout -dates -subject -ext subjectAltName

On nginx, ssl_certificate must point to the full chain file (for Let’s Encrypt: fullchain.pem, not cert.pem). Full steps: Fix: No response › SSL/TLS error.

Platform steps

Cloudflare

In the Cloudflare dashboard › your site › SSL/TLS › Overview: with the mode set to Full (strict) (SSL/TLS modes explained), the origin certificate must be valid too. Edge certificates renew automatically.

nginx

ssl_certificate     /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
ssl_protocols       TLSv1.2 TLSv1.3;

Verify the fix

openssl s_client shows Verify return code: 0 (ok) for both the www and non-www names. Then Crawl again.

Automation

Automatable on most hosts (AutoSSL, certbot): needs hosting panel or server SSH access.

Connection refused or reset

How we detect it

The server refused the connection, or closed it mid-way. The Pages tab shows No response.

Why it matters

A firewall that drops data-centre traffic can also drop SEO tools, uptime monitors and sometimes search engine crawlers that use similar networks.

Common causes

• If it fails for us but works for you → a firewall (server firewall, host DDoS protection, country blocking) blocks our IP.
• If it fails for everyone → the web server is down or port 443 is closed.
• If DNS points to an old server → the old server refuses connections.

Fix: basic

1. Check the site loads in your browser.
2. Allowlist 147.93.19.85 and 2a02:4780:12:d33e::1 in every firewall in front of the site: CDN (Cloudflare), host firewall (server firewalls), security plugin (WordPress plugins).
3. If the site uses country blocking (geo-blocking), add our IPs as an exception. Our server is in a data centre that may be outside the client’s country.
4. Click Crawl again.

Fix: expert

Add our IPs to the server firewall’s allow list (CSF, fail2ban, iptables). Copy-paste lines for each firewall are in Fix: 403 and 429 › Host firewall blocks our IP and Fix: No response › Connection refused or reset.

Platform steps

cPanel with CSF

Plugins › ConfigServer Security & Firewall › Firewall Allow IPs (csf.allow), add 147.93.19.85 # SEOAuditSoftware and 2a02:4780:12:d33e::1 # SEOAuditSoftware, then Restart csf+lfd.

Verify the fix

The Pages tab shows real status codes (200, 301, 404) instead of No response after Crawl again.

Automation

Assisted: needs server SSH or hosting panel access.

403 from a firewall or WAF

How we detect it

The server answered with HTTP 403 Forbidden. In a crawl the home page shows status 403. Classic audits show Audit failed: the site blocks automated page downloads (403). GEO and Local audits show The page returned HTTP 403.

Why it matters

A 403 means the site decided to refuse our request. The same rules sometimes block real search engines, link previews or uptime tools.

Common causes

• If the response has server: cloudflare and a cf-ray header → Cloudflare (see the next section).
• If the page says “Your access to this site has been limited by the site owner” → Wordfence.
• If it says “Access Denied - Sucuri Website Firewall” → Sucuri.
• If it says “Not Acceptable!” or “Mod_Security” → ModSecurity on the server.
• If it is a plain nginx or Apache 403 page → a User-Agent or IP rule in the server config or .htaccess.
• If it mentions Imunify360 or “Access denied by Imunify360 bot-protection” → Imunify360 on a cPanel host.

Fix: basic

1. Find which tool blocks: run step 6 of Quick triage and match the block page text above.
2. Open that tool and add 147.93.19.85 and 2a02:4780:12:d33e::1 to its allow list. Steps per tool: Wordfence, Sucuri, ModSecurity, Imunify360, nginx or Apache / .htaccess.
3. If the tool allows User-Agent rules, also allow SEOAuditSoftware.
4. Click Crawl again or Run again.

Fix: expert

For server rules, add an IP exception before any block rule. Example for Apache 2.4 .htaccess:

# Let SEO Audit Software through bot rules that use the "bad_bot" variable
SetEnvIf Remote_Addr "^147\.93\.19\.85$" allow_seoaudit
SetEnvIfNoCase User-Agent "SEOAuditSoftware" allow_seoaudit
SetEnvIf allow_seoaudit 1 !bad_bot

Platform steps

WordPress

Wordfence, Solid Security, Sucuri, All-In-One Security and LiteSpeed/QUIC.cloud steps: Fix: WordPress security plugins block the crawl.

nginx, Apache, cPanel, Plesk, hPanel

ModSecurity, CSF, Imunify360 and server rules: Fix: 403 and 429 from the server, host firewall or ModSecurity.

Verify the fix

Run the curl command from Quick triage (status 200), then Crawl again: the home page shows 200 and more pages are found.

Automation

Assisted: needs WordPress admin (plugin settings), hosting panel or server SSH, depending on the tool that blocks.

Cloudflare bot challenge

How we detect it

The home page returns 403 (sometimes 503) with the headers server: cloudflare and cf-ray, and the HTML contains “Just a moment…”, “Attention Required!” or “Error 1020 Access denied”. Our crawler can’t run the challenge JavaScript, so it never gets the page.

Why it matters

Challenge pages are shown to automated traffic that Cloudflare doesn’t recognise. They also block many SEO, monitoring and preview tools.

Common causes

• If you see “Just a moment…” → Bot Fight Mode, Under Attack mode, a high Security Level or a custom rule with a Managed Challenge.
• If you see “Error 1020” → a WAF custom rule blocks us.
• If you see “Error 1015” or status 429 → a rate limiting rule.
• If you see “Error 1010” → Browser Integrity Check flagged the request.

Fix: basic

1. Log in to the Cloudflare dashboard and select the domain.
2. Turn off Under Attack mode if it is on (what Under Attack mode does).
3. Create a WAF custom rule that skips security for our IPs (full steps and expression in the Cloudflare guide).
4. On the Free plan, turn off Bot Fight Mode under Security › Bots: custom rules can’t skip it (Bot Fight Mode docs).
5. Click Crawl again.

Fix: expert

Custom rule expression (in the Cloudflare dashboard › your site › Security › Security rules › Custom rules, action Skip; see Cloudflare custom rules):

(ip.src eq 147.93.19.85) or (ip.src eq 2a02:4780:12:d33e::1) or (http.user_agent contains "SEOAuditSoftware")

Every setting, plan difference and menu path: Fix: Cloudflare blocks the crawl or audit.

Platform steps

Cloudflare

See the Cloudflare guide for Bot Fight Mode, Super Bot Fight Mode, Security Level, Browser Integrity Check and rate limiting rules.

Verify the fix

In the Cloudflare dashboard, open Security › Events (or Security › Analytics › Events) and filter by IP 147.93.19.85: new requests show the action Skip. Then Crawl again.

Automation

Automatable: needs a Cloudflare API token with Zone WAF edit rights to create the skip rule.

429 rate limit

How we detect it

Pages return HTTP 429 Too Many Requests. Often the first pages are fine and later ones get 429.

Why it matters

Our crawler is polite (one request at a time, a pause between pages), so a limit that blocks it is very strict and may also slow Googlebot.

Common causes

• If the 429 page is from Cloudflare (Error 1015) → a Cloudflare rate limiting rule.
• If it is from WordPress → Wordfence rate limiting or a similar plugin.
• If it is from nginx → limit_req in the server config.
• If the site is on Shopify → Shopify’s own limits (can’t be changed; see the platform guide).

Fix: basic

1. Find the tool that sends the 429 (look at the server header and the page text).
2. Add our IPs as an exception in that tool, or raise the limit to at least 60 requests per minute per IP. Steps: Cloudflare, Wordfence, nginx and other servers.
3. Click Crawl again.

Fix: expert

nginx: exempt our IPs from limit_req with a geo block. Full config: Fix: 403 and 429 › 429 Too Many Requests.

Platform steps

Cloudflare

In the Cloudflare dashboard › your site › Security › Security rules › Rate limiting rules (rate limiting docs): add and not ip.src in {147.93.19.85 2a02:4780:12:d33e::1} to the rule expression, or skip rate limiting in the custom skip rule.

WordPress (Wordfence)

Wordfence › Firewall › All Firewall Options › Rate Limiting: set “If a crawler’s page views exceed” to Unlimited or a higher value, and allowlist our IPs (Wordfence rate limiting docs).

Shopify, Wix, Squarespace

See Fix: Crawling Shopify, Wix, Squarespace and Webflow sites.

Verify the fix

After Crawl again, the Pages tab has no 429 rows.

Automation

Assisted: needs access to the tool that limits (Cloudflare API token, WordPress admin or server SSH).

5xx errors

How we detect it

Pages return 500, 502, 503 or 504. They are listed in the Server errors (5xx) issue and in the Pages tab under the 5xx chip.

Why it matters

Google treats repeated 5xx errors as a sign the site is unhealthy, slows crawling and can drop pages from the index if the errors last.

Common causes

• If 500 → a PHP fatal error, a broken plugin or theme, a bad .htaccess, or out of memory.
• If 502 → PHP-FPM or the app server crashed or ran out of workers.
• If 503 → maintenance mode (a leftover WordPress .maintenance file) or host resource limits.
• If 504 → the page takes too long to build and the proxy gives up.

Fix: basic

1. In Site crawl, open the crawl › Issues › Server errors (5xx), and open each failing URL in your browser.
2. If it says “Briefly unavailable for scheduled maintenance”, delete the .maintenance file in the site root (File Manager in the hosting panel).
3. In WordPress, update or deactivate the plugin named in the error or recovery-mode email.
4. Click Crawl again.

Fix: expert

Turn on the WordPress debug log in wp-config.php and read wp-content/debug.log:

define( 'WP_DEBUG', true );
define( 'WP_DEBUG_LOG', true );
define( 'WP_DEBUG_DISPLAY', false );

Every code (500, 502, 503, 504), with nginx, Apache and PHP limits: Fix: Server errors (5xx) — 500, 502, 503, 504.

Platform steps

Cloudflare

Errors 520–526 come from Cloudflare when the origin fails; fix the origin server.

Verify the fix

Each URL opens with status 200. After Crawl again, Compare shows Server errors (5xx) as fixed.

Automation

Assisted: needs server SSH or hosting panel access (logs, PHP settings) and WordPress admin.

Redirects to another host or redirect loops

How we detect it

The start page redirects to a different domain, so the crawl stops after one page. Or the URL redirects more than 5 times (a loop) and shows No response.

Why it matters

Redirect loops break the site for everyone. Redirects to another domain mean the client’s website in the app is not the real site.

Common causes

• If example.com redirects to shop.example.net → the real site is on another domain.
• If it redirects to a country domain (example.de) → geo-redirects based on our server’s location.
• If http and https keep sending you back and forth → Cloudflare Flexible SSL with an HTTPS redirect on the server.
• If WordPress loops → the WordPress Address and Site Address don’t match the real URL.

Fix: basic

1. Run the redirect command from Quick triage to see the final URL.
2. If the site moved to another domain, open the client in Clients and edit it to set the final domain as the website.
3. For subdomains such as blog.example.com, turn on Include subdomains when you start the crawl.
4. For loops on Cloudflare, in the Cloudflare dashboard › your site, set SSL/TLS › Overview to Full (strict) (details).
5. Click Crawl again.

Fix: expert

See every case with code in Fix: Crawl finds few pages › Redirects to another host and Redirect loops.

Platform steps

WordPress

In WordPress admin, Settings › General (/wp-admin/options-general.php): WordPress Address (URL) and Site Address (URL) must both be the same final https URL.

Verify the fix

The redirect check ends on the client’s own domain in 1 redirect or fewer. Crawl again finds more pages.

Automation

Assisted: needs Cloudflare, WordPress admin or server access depending on the source of the redirect.

robots.txt blocking

How we detect it

The crawler reads /robots.txt first. URLs disallowed for SEOAuditSoftware (or for * when there is no group for us) are skipped silently: they don’t appear in the Pages tab, so compare missing sections with the Disallow lines in robots.txt.

Why it matters

A Disallow: / meant for a staging site often stays after launch and also blocks Google.

Common causes

• If robots.txt has User-agent: * and Disallow: / → everything is blocked for all bots.
• If it has a group for a list of SEO tools → we may be named or covered.

Fix: basic

1. Open https://example.com/robots.txt in your browser.
2. Remove Disallow: / under User-agent: * if the site should be public.
3. Or add an allow group for us (snippet below).
4. Click Crawl again.

Fix: expert

User-agent: SEOAuditSoftware
Allow: /

More cases (WordPress, Shopify, Wix): Fix: Crawl finds few pages › robots.txt blocking. Google’s rules for the file: robots.txt introduction.

Platform steps

WordPress (Yoast SEO)

In WordPress admin, Yoast SEO › Tools › File editor lets you edit robots.txt (Yoast help).

Verify the fix

https://example.com/robots.txt shows the new rules, and the robots.txt report in Search Console › Settings shows the new version for Google. Crawl again finds the pages.

Automation

Automatable: needs WordPress admin (Yoast or Rank Math) or file access to edit robots.txt.

Login wall

How we detect it

The page returns 401, or redirects to a login page, or shows a password form. The crawler can’t sign in.

Why it matters

Google can’t see pages behind a login either. Only public pages can rank.

Common causes

• If the whole site asks for a password → a staging site, a maintenance or launch-page plugin, or platform password protection.
• If a browser pop-up asks for a username → HTTP basic authentication.

Fix: basic

1. Crawl the public live site, not the staging site.
2. Turn off site-wide password protection when the site is live.
3. Click Crawl again.

Fix: expert

To crawl a protected staging site, let our IP in without a password. Snippets for Apache and nginx: Fix: Crawl finds few pages › Login wall.

Platform steps

Shopify

In Shopify admin › Online Store › Preferences › Password protection (Shopify help): untick Restrict access to visitors with the password.

Verify the fix

The home page returns 200 without a login. Crawl again.

Automation

Manual: needs a decision by the site owner; then WordPress admin, platform admin or server access.

JavaScript-rendered links

How we detect it

The start page returns 200 but the crawl finds only that page or very few, because the menu and links are added by JavaScript.

Why it matters

Google can render JavaScript, but later and less reliably. Plain HTML links get found faster by every crawler, including AI crawlers that don’t run JavaScript.

Common causes

• If the site is built with React, Vue, Angular or a page builder that loads menus with JavaScript → links aren’t in the HTML.
• If navigation uses buttons with click handlers instead of <a href> links → there are no links to follow.

Fix: basic

1. In Site crawl, start a new crawl with Render JavaScript on. It renders the start page and finds the links added by JavaScript there.
2. Add a plain HTML footer menu or an HTML sitemap page linking to the main sections.

Fix: expert

Render links on the server (server-side rendering or pre-rendering) and use real anchors:

<a href="/services/">Services</a>

Details: Fix: Crawl finds few pages › JavaScript-rendered links and Google’s JavaScript SEO basics.

Platform steps

Next.js / Nuxt

Use the framework’s Link component (it outputs a real <a href>), and server-side or static rendering for content pages.

Verify the fix

View the page source (Ctrl+U): the menu links appear as <a href> in the HTML. Crawl again finds more pages.

Automation

Manual: needs code changes by the site’s developer or theme settings.

Common questions

Related articles

• Fix: Cloudflare blocks the crawl or audit (403, 'Just a moment', 429)
• Fix: WordPress security plugins block the crawl (Wordfence, Solid Security, Sucuri, LiteSpeed)
• Fix: 403 and 429 from the server, host firewall or ModSecurity (nginx, Apache, cPanel, Plesk, hPanel)
• Fix: Server errors (5xx) — 500, 502, 503, 504
• Fix: No response — timeouts, DNS, SSL/TLS and connection errors
• Fix: Crawl finds few pages — redirects, robots.txt, login walls and JavaScript links
• Fix: Crawling Shopify, Wix, Squarespace and Webflow sites
• Crawl a whole website
• An audit failed or did not finish

Still need help?

Ask in the chat inside the app, or send us a message on the contact page. You can also browse all help topics.