Fix: Crawling Shopify, Wix, Squarespace and Webflow sites

On hosted platforms you can’t edit the server, firewall or bot protection, so you can’t allowlist our IP. You can still fix the settings that block most crawls: password pages, visibility, robots.txt and redirects. This guide lists what works on each platform.

What to allowlist (where a platform allows it): IPv4 147.93.19.85, IPv6 2a02:4780:12:d33e::1, User-Agent token SEOAuditSoftware. Details: What to allowlist.

Shopify

How we detect it

Typical signs in the crawl: the home page shows the store’s password page (status 200 or 401 with very little text, 1 page crawled); 429 status codes after some pages; a 403 with server: cloudflare (Shopify’s own edge); or the home page redirects to another country domain (example.com → example.de).

Why it matters

A password page or a forced geo-redirect hides the real store from Google too. Rate limits mostly affect tools, not Googlebot, but they cut the crawl short.

Common causes

• If you see “Opening soon” or “Enter store using password” → Password protection is on.
• If pages work and then return 429 → Shopify’s rate limiting for automated traffic.
• If you get a 403 from the first request → Shopify’s bot protection flagged our data-centre IP, or an app (for example a bot-blocking or anti-fraud app) blocks it.
• If the home page redirects to another domain or /en-de/ → Markets automatic redirection or a geolocation app sends our server to another market.
• If many crawled URLs contain ?filter. or ?sort_by= → collection filters eat the page budget.

Fix: basic

Remove the password page (when the store is live)

1. In Shopify admin, go to Online Store › Preferences (help).
2. Under Password protection, untick Restrict access to visitors with the password.
3. Click Save.

Stop geo-redirects for the crawler

1. In Shopify admin, go to Settings › Markets.
2. Open Preferences (or the … menu on the Markets page) and turn off automatic redirection based on visitor location.
3. If a geolocation app redirects (for example Shopify’s Geolocation app or a third-party one), open the app and switch it from redirect to banner / pop-up recommendation mode.
4. Click Save.

Rate limits and 403s

1. Start the crawl in Site crawl with Max pages 500.
2. Run it outside the store’s busiest hours.
3. If it still stops, click Crawl again later; blocks on Shopify are usually temporary.
4. Disable bot-blocking apps temporarily (Shopify admin › Apps › the app › its settings) to test whether an app causes the 403.
5. For the most important pages (home, top collections, top products), run single-page audits as well in Audits. See Run an audit.

Fix: expert

Add an allow group for our crawler in robots.txt.liquid, keeping Shopify’s defaults:

1. In Shopify admin, go to Online Store › Themes, click … next to the live theme, then Edit code (Shopify: editing robots.txt.liquid).
2. Under Templates, click Add a new template, choose robots.txt, click Done.
3. Replace the content with:

User-agent: SEOAuditSoftware
Disallow: /cart
Disallow: /checkout
Disallow: /account
Disallow: /*?*filter.
Disallow: /*?*sort_by=

{% for group in robots.default_groups %}
  {{- group.user_agent }}
  {%- for rule in group.rules %}
    {{ rule }}
  {%- endfor %}
  {%- if group.sitemap != blank %}
    {{ group.sitemap }}
  {%- endif %}
{% endfor %}

4. Click Save.

This keeps the crawler out of cart, account and filter URLs so the page budget goes to real pages.

Platform steps

Shopify

All steps above. Shopify’s edge (Cloudflare) can’t be configured by store owners, so the Cloudflare guide doesn’t apply unless the store has its own Cloudflare account in front of Shopify.

Verify the fix

1. curl -sS -o /dev/null -w "%{http_code} %{url_effective}\n" -L https://example.com/ returns 200 on the store’s own domain.
2. https://example.com/robots.txt shows the SEOAuditSoftware group.
3. Site crawl › Crawl again: products and collections appear in the Pages tab, without 429 rows.

Automation

Assisted: needs Shopify admin (Online Store and Markets settings) or a Shopify Admin API token with theme write access for robots.txt.liquid.

Wix

How we detect it

The crawl gets the Wix “site not published” or “Connect your domain” page, a members-only login, or (rarely) a 403/429 from Wix’s servers.

Why it matters

Unpublished or members-only pages are invisible to Google as well.

Common causes

• If you see Wix’s “This site isn’t published” page → the site isn’t published, or the client’s domain isn’t connected.
• If pages show a login → the pages are set to Members only or Password holders.
• If everything is Noindex → Let search engines index your site is off.
• If you get a 403 or 429 → Wix’s own protection; there’s no setting for it.

Fix: basic

Publish and connect the domain

1. In the Wix editor, click Publish.
2. In the Wix dashboard, go to Settings › Domains and make sure the client’s domain is connected and primary.

Make pages public

1. In the editor, open Pages & Menu, click … next to a page › Permissions.
2. Choose Everyone, then Publish.

Allow indexing

1. In the Wix dashboard, go to Marketing & SEO › SEO (newer dashboards: Settings › SEO).
2. Open SEO Settings and switch on Let search engines index your site.

403 or 429 from Wix

1. In Site crawl, click Crawl again later with Max pages 500.
2. If it keeps failing, send Wix support these exact lines: “Our SEO audit crawler (User-Agent token SEOAuditSoftware, IP 147.93.19.85) gets HTTP 403/429 on example.com. Please check whether bot protection blocks it.”

Fix: expert

Edit robots.txt only if it blocks pages you need:

1. Go to Marketing & SEO › SEO › Go to Robots.txt Editor (or SEO Tools › Robots.txt Editor).
2. Click View File, add the group below at the top, click Save Changes.

User-agent: SEOAuditSoftware
Allow: /

Platform steps

Wix

Wix renders pages on its servers, so links are in the HTML and Render JavaScript usually isn’t needed.

Verify the fix

The home page returns 200 with the real content; Crawl again finds the site’s pages.

Automation

Manual: needs Wix dashboard and editor access.

Squarespace

How we detect it

The crawl gets Squarespace’s lock screen (“This site is private” or a password form), or every page is Noindex, or pages are missing because robots.txt blocks them.

Why it matters

Private and password-protected sites can’t be crawled or indexed. Hiding a site from search engines removes it from Google.

Common causes

• If you see a lock screen → Site availability is Private or Password protected (default on trials).
• If pages are Noindex → the site or the page is hidden from search engines in SEO settings.
• If the site is public but the crawl is blocked → the Crawlers settings block search engine crawlers or a page-level password is set.

Fix: basic

Make the site public

1. In your Squarespace account, open the site and go to Settings › Site availability (newer menus: Settings › Website › Site availability).
2. Choose Public and click Save. Trial sites must be upgraded to a paid plan to go public.

Search engine crawler settings

1. Go to Settings › Crawlers (newer menus: Settings › Website › Crawlers).
2. Turn off Block search engine crawlers.
3. Block known artificial intelligence crawlers affects AI crawlers (GPTBot and similar), not our crawler. Leave it as the client prefers; note that it lowers the GEO audit’s AI crawlers allowed in robots.txt result (see Fix: GEO and AI checks).

Page-level settings

1. Open Pages, click the gear next to a page › SEO.
2. Untick Hide this page from search engine results for pages that should rank.
3. In the page’s General tab, remove any page password.

Fix: expert

Squarespace doesn’t allow editing robots.txt directly. The settings above control it (Squarespace help). Check the result:

curl -sS https://example.com/robots.txt | head -n 40

Platform steps

Squarespace

Squarespace renders pages on its servers, so normal crawls see the links.

Verify the fix

The home page returns 200 without a lock screen, robots.txt has no Disallow: / for *, and Crawl again finds the pages.

Automation

Manual: needs Squarespace admin access.

Webflow

How we detect it

Every page shows Noindex (crawling the .webflow.io staging domain), the site shows a password form, or the custom domain redirects in a loop.

Why it matters

The staging domain is meant to stay out of Google; crawling it gives a misleading report. A password form hides the whole site.

Common causes

• If the URL ends in .webflow.io → you’re crawling the staging domain, which is noindex by default.
• If you see “This page is password protected” → site or page password protection.
• If the custom domain loops → a Cloudflare proxy in front of Webflow with the wrong SSL mode, or both Cloudflare and Webflow forcing redirects.

Fix: basic

Crawl the custom domain

1. Open Clients, open the client › Settings tab, and make sure the website is the custom domain (for example example.com), not example.webflow.io.
2. In the Webflow dashboard, open the site › Site settings › Publishing and check the custom domain is published and set as the default domain.

Password protection

1. Go to Site settings › General and find Password protection (some workspaces show it under Site settings › Publishing).
2. Turn off site-wide protection.
3. For single pages: open the page’s Settings in the Designer and remove the page password. Publish.

Indexing settings

1. Go to Site settings › SEO › Indexing.
2. Leave Disable Webflow subdomain indexing on (it only affects .webflow.io).
3. Check that the custom domain has no noindex added through custom code (Site settings › Custom code).

Fix: expert

If Cloudflare proxies a Webflow site, in the Cloudflare dashboard › your site, set SSL/TLS › Overview to Full (strict) (see Redirect loop with Flexible SSL) and remove extra http→https redirect rules in Cloudflare; Webflow already redirects to https. Check:

curl -sS -I -L --max-redirs 10 https://example.com/ 2>&1 | grep -iE "^HTTP|^location"

Platform steps

Webflow

Webflow publishes static HTML with real links, so crawls normally work without Render JavaScript.

Verify the fix

The custom domain returns 200 with at most one redirect, pages show Yes under Indexable, and Crawl again finds all published pages.

Automation

Manual: needs Webflow site settings access (and Cloudflare access if it is in front).

Other hosted builders (GoDaddy, Hostinger Website Builder, Weebly, Duda)

How we detect it

The same signs as above: a lock or “coming online” page, 403/429 with no settings to change, or noindex everywhere.

Why it matters

These builders control the server, so only their own settings can change access.

Common causes

• If the site is unpublished or private → visibility settings.
• If you get 403/429 → the builder’s bot protection.

Fix: basic

1. Publish the site and connect the domain in the builder’s dashboard.
2. Turn off site-wide password protection.
3. In the builder’s SEO settings, allow search engines to index the site.
4. Click Crawl again in Site crawl. If the 403/429 stays, send the builder’s support this text: “Our SEO audit crawler (User-Agent token SEOAuditSoftware, IPs 147.93.19.85 and 2a02:4780:12:d33e::1) gets HTTP 403/429 on example.com. Please allow it.”

Fix: expert

No server access is available on these builders. Check the response:

curl -sS -I -A "SEOAuditSoftware" https://example.com/ | grep -iE "^HTTP|^server"

Platform steps

Hostinger Website Builder

In hPanel › Websites › Manage › Website Builder › Edit website › Settings › SEO settings: allow indexing; publish the site.

Verify the fix

curl returns 200, then Crawl again finds the pages.

Automation

Manual: needs the builder’s admin access.

Common questions

Related articles

• Fix: Site can't be crawled (server errors, blocks and timeouts)
• Fix: Crawl finds few pages — redirects, robots.txt, login walls and JavaScript links
• Fix: URLs with parameters and URL structure
• Fix: Cloudflare blocks the crawl or audit (403, 'Just a moment', 429)
• Crawl a whole website
• An audit failed or did not finish

Still need help?

Ask in the chat inside the app, or send us a message on the contact page. You can also browse all help topics.