Fix: Cloudflare blocks the crawl or audit (403, 'Just a moment', 429)

Cloudflare sits in front of many client sites. Its bot protection sees our crawler as unknown automated traffic and blocks or challenges it. This guide fixes that with one rule and a few settings.

What to allowlist: IPv4 147.93.19.85, IPv6 2a02:4780:12:d33e::1, and the User-Agent token SEOAuditSoftware (Site crawl, GEO and Local SEO audits). Classic audits use a normal Chrome User-Agent, so always include the IPs. Full details: What to allowlist.

Is it Cloudflare?

1. Run this from a terminal (replace example.com):

curl -sS -I -A "SEOAuditSoftware" https://example.com/ | grep -iE "^HTTP|^server|^cf-ray|^cf-mitigated"

2. Cloudflare is in front if you see server: cloudflare and a cf-ray header.
3. cf-mitigated: challenge means a challenge page was served.
4. To see the page text:

curl -sS -A "SEOAuditSoftware" https://example.com/ | grep -oE "Just a moment|Attention Required|Error 10[0-9]{2}|Access denied|Sorry, you have been blocked" | sort -u

Cloudflare bot challenge (“Just a moment…”)

How we detect it

The home page returns 403 or 503 with server: cloudflare, and the HTML contains “Just a moment…”. Our crawler doesn’t run challenge JavaScript, so it gets no page. The crawl finds 1 page or shows a 403; classic audits fail with Audit failed: the site blocks automated page downloads (403).

Why it matters

While the challenge is in place, no crawl or audit can read the site. The same settings often block other useful tools (uptime monitors, link previews).

Common causes

• If the whole site challenges everyone, including you in a private window → Under Attack mode is on.
• If only automated tools are challenged and the zone is on the Free plan → Bot Fight Mode.
• If the zone is on Pro or higher → Super Bot Fight Mode (“Definitely automated” set to Managed Challenge or Block).
• If none of these → Security Level set to High, or a custom rule with the action Managed Challenge.

Fix: basic

Do these in order and test after each.

Step 1: Under Attack mode off

1. Log in to the Cloudflare dashboard and select the domain.
2. Go to Security › Settings and find I’m Under Attack mode (older dashboard: Overview › Quick Actions › Under Attack Mode).
3. Switch it off. What it does: Under Attack mode (Cloudflare docs).

Step 2: create the skip rule (see the next section, WAF custom rule).

Step 3: Bot Fight Mode (Free plan)

1. In the Cloudflare dashboard, select the domain, go to Security › Settings and filter by Bot traffic (older dashboard: Security › Bots).
2. Switch Bot Fight Mode off.
3. Custom rules and IP Access rules can’t skip Bot Fight Mode, so this is the only way on the Free plan. See Bot Fight Mode (Cloudflare docs).

Step 4: Super Bot Fight Mode (Pro, Business, Enterprise)

1. In the Cloudflare dashboard, select the domain and go to Security › Settings › Bot traffic (older dashboard: Security › Bots › Configure Super Bot Fight Mode). See Super Bot Fight Mode (Cloudflare docs).
2. Your skip rule from Step 2 can skip Super Bot Fight Mode, so you can keep it on.
3. If you prefer, set Definitely automated to Allow instead.

Step 5: Security Level

1. In the Cloudflare dashboard, select the domain, go to Security › Settings and find Security Level (older dashboard: Security › Settings › Security Level).
2. Set it to Medium or lower. High challenges many data-centre IPs.

Fix: expert

Use the API to create the skip rule (see WAF custom rule below) so it’s identical on every client zone. Keep Bot Fight Mode off on Free zones; on paid zones, skip Super Bot Fight Mode in the rule.

Platform steps

Cloudflare

All steps are above. If the domain is on a Cloudflare for SaaS or a host-managed Cloudflare (for example a host’s “CDN” switch), the host’s panel may hide these settings: in that case use the host’s own bot protection switch (for Hostinger: hPanel › Websites › Manage › Performance › CDN, turn off bot protection or the security option there).

WordPress

The Cloudflare WordPress plugin can turn on Under Attack mode too. In your WordPress admin (/wp-admin/) › Settings › Cloudflare, check that it is off.

Verify the fix

1. Run the curl commands from Is it Cloudflare? again: status 200, no cf-mitigated header.
2. In the Cloudflare dashboard, select the domain and open Security › Events (older: Security › Events under Analytics; see Security Events docs), add the filter Source IP equals 147.93.19.85.
3. New requests show the action Skip or no action. No Managed Challenge rows.
4. In the app, click Crawl again in Site crawl or Run again in Audits.

Automation

Automatable: needs a Cloudflare API token with Zone › Zone Settings: Edit, Zone › Zone WAF: Edit and Zone › Bot Management: Edit permissions.

WAF custom rule (Error 1020 and Error 1010)

How we detect it

The response is 403 with Error 1020 “Access denied” (a custom or firewall rule), or Error 1010 (Browser Integrity Check flagged the request).

Why it matters

A blocking rule meant for spam or attacks catches our crawler. One skip rule fixes it for every security product in Cloudflare that supports skipping.

Common causes

• If there is a rule blocking “known bots”, data-centre ASNs, countries or user agents containing “bot” or “crawler” → it matches us.
• If Browser Integrity Check is on → requests without typical browser headers can get Error 1010.
• If there is an old User Agent Blocking rule → it may match parts of our User-Agent.

Fix: basic

Create the skip rule:

1. In the Cloudflare dashboard, select the domain and go to Security › Security rules (older dashboard: Security › WAF › Custom rules). Reference: WAF custom rules (Cloudflare docs).
2. Click Create rule › Custom rules.
3. Rule name: Allow SEO Audit Software.
4. Click Edit expression and paste the expression from Fix: expert below.
5. Choose action: Skip.
6. Under WAF components to skip, tick All remaining custom rules, All rate limiting rules, All managed rules and All Super Bot Fight Mode rules (shown on paid plans).
7. Under More components to skip, tick Browser Integrity Check, Security Level, User Agent Blocking and Zone Lockdown.
8. Keep Log matching requests on so you can check it later.
9. Under Place at, choose First.
10. Click Deploy.

Fix: expert

Expression to paste:

(ip.src eq 147.93.19.85) or (ip.src eq 2a02:4780:12:d33e::1) or (http.user_agent contains "SEOAuditSoftware")

IP-only version (safest; works for crawls and every audit type):

(ip.src in {147.93.19.85 2a02:4780:12:d33e::1})

Create it with the API (replace ZONE_ID and the token; this adds the rule to the zone’s custom rules entry point):

curl -sS "https://api.cloudflare.com/client/v4/zones/ZONE_ID/rulesets/phases/http_request_firewall_custom/entrypoint" \
  -H "Authorization: Bearer CF_API_TOKEN" | head -c 400
# Use the ruleset id from the reply as RULESET_ID:
curl -sS -X POST "https://api.cloudflare.com/client/v4/zones/ZONE_ID/rulesets/RULESET_ID/rules" \
  -H "Authorization: Bearer CF_API_TOKEN" -H "Content-Type: application/json" \
  --data '{"description":"Allow SEO Audit Software","expression":"(ip.src in {147.93.19.85 2a02:4780:12:d33e::1}) or (http.user_agent contains \"SEOAuditSoftware\")","action":"skip","action_parameters":{"ruleset":"current","phases":["http_ratelimit","http_request_firewall_managed","http_request_sbfm"],"products":["bic","securityLevel","uaBlock","zoneLockdown"]},"logging":{"enabled":true},"position":{"index":1}}'

If the zone has no custom rules yet, the first command returns “not found”: create the rule once in the dashboard instead.

Alternative: IP Access rule (Cloudflare docs). In the Cloudflare dashboard, go to Security › WAF › Tools (newer dashboard: Security › Security rules › IP access rules), enter 147.93.19.85, choose Allow, This website, click Add. Repeat for 2a02:4780:12:d33e::1. IP Access Allow skips most security features but not Bot Fight Mode.

Platform steps

Cloudflare

To find which rule blocked us: Cloudflare dashboard › your site › Security › Events, filter Source IP equals 147.93.19.85, open a blocked event and read Rule ID and Service. Edit or add an exception to that rule.

Verify the fix

Security › Events shows our requests with action Skip from the rule Allow SEO Audit Software. curl returns 200. Then Crawl again in Site crawl.

Automation

Automatable: needs a Cloudflare API token with Zone WAF: Edit.

Rate limit (429, Error 1015)

How we detect it

Pages return HTTP 429 or Cloudflare Error 1015 “You are being rate limited”, often after the first few pages.

Why it matters

Our crawler sends one request at a time with a 0.25 second pause, so a rule that blocks it is tight enough to hurt other good bots too.

Common causes

• If a rate limiting rule counts all requests per IP over 10 seconds with a low threshold → our crawl crosses it.
• If the rule protects /wp-login.php or /xmlrpc.php only → it shouldn’t affect us; look elsewhere.

Fix: basic

1. In the Cloudflare dashboard, select the domain, go to Security › Security rules and open the Rate limiting rules list (older: Security › WAF › Rate limiting rules). Reference: Rate limiting rules (Cloudflare docs).
2. Open the rule that matched (check Security › Events for “Rate limiting rules” events from our IP).
3. Add our IPs as an exception (snippet below), or raise the threshold to at least 60 requests per minute.
4. Click Save.

Fix: expert

Wrap the rule’s existing expression in brackets and add the IP exception. For example, if the rule was (http.request.uri.path wildcard "/*"), change it to:

(http.request.uri.path wildcard "/*") and not (ip.src in {147.93.19.85 2a02:4780:12:d33e::1})

Keep your existing conditions and add only the and not (ip.src in {...}) part. If you created the skip rule above with All rate limiting rules ticked, this is already covered.

Platform steps

Cloudflare

Rate limiting rules run after custom rules, so the skip rule placed First covers them.

Verify the fix

Crawl again in Site crawl: no 429 rows in the Pages tab. Security › Events shows no rate limiting actions for our IP.

Automation

Automatable: needs a Cloudflare API token with Zone WAF: Edit.

Redirect loop with Flexible SSL

How we detect it

The URL redirects more than 5 times and the Pages tab shows No response. In a browser: “This page isn’t working, redirected you too many times”.

Why it matters

A loop breaks the site for every visitor and every crawler, not only for us.

Common causes

• If SSL/TLS mode is Flexible and the server (or WordPress) redirects http to https → Cloudflare talks to the server over http, the server redirects to https, and it repeats forever.

Fix: basic

1. Make sure the origin server has a valid certificate (most hosts have free Let’s Encrypt or AutoSSL). Check it with the SSL Labs test.
2. In the Cloudflare dashboard, select the domain, go to SSL/TLS › Overview and choose Full (strict) (SSL/TLS modes explained) (use Full only if the origin certificate is self-signed).
3. Go to SSL/TLS › Edge Certificates and turn on Always Use HTTPS.

Fix: expert

Test the loop:

curl -sS -o /dev/null -L --max-redirs 10 -w "%{num_redirects} redirects, final: %{url_effective}\n" http://example.com/

Platform steps

WordPress

In your WordPress admin (/wp-admin/options-general.php) › Settings › General: both URLs must start with https://. More redirect fixes: Fix: HTTPS, SSL and redirects.

Verify the fix

The curl command shows 1 redirect or fewer and a 200 final URL. Crawl again in Site crawl.

Automation

Automatable: needs a Cloudflare API token with Zone Settings: Edit.

Cloudflare Access login or Turnstile

How we detect it

Every page redirects to *.cloudflareaccess.com or shows a Cloudflare Access sign-in, or a form with a Turnstile widget blocks the content.

Why it matters

Our crawler can’t sign in or solve Turnstile. Content behind them is invisible to search engines too.

Common causes

• If the redirect goes to cloudflareaccess.com → a Zero Trust Access application protects the site (common on staging sites).
• If only one form has Turnstile → that is fine: it doesn’t block page crawling.

Fix: basic

1. Crawl the public live site instead of the protected one.
2. If you must crawl the protected site, open the Cloudflare dashboard › Zero Trust › Access › Applications, open the application, and add a policy with the action Bypass and the include rule IP ranges 147.93.19.85/32 and 2a02:4780:12:d33e::1/128.
3. Save and put the Bypass policy first.

Fix: expert

Remove Access from production hostnames. Keep it only on staging hostnames, with the IP Bypass policy above.

Platform steps

Cloudflare

Zero Trust › Access › Applications › (application) › Policies › Add a policy.

Verify the fix

curl to the home page returns 200 instead of a redirect to cloudflareaccess.com. Crawl again in Site crawl.

Automation

Assisted: needs Cloudflare Zero Trust admin access.

AI crawler blocking settings

Cloudflare’s Block AI bots setting and AI Crawl Control target AI crawlers such as GPTBot. They don’t target our User-Agent. Our GEO audit’s AI crawlers allowed in robots.txt check (fix guide) reads robots.txt, but Cloudflare’s setting can block those AI crawlers at the edge even when robots.txt allows them. To let AI crawlers in, open the Cloudflare dashboard › your site › Security › Settings (or AI Crawl Control) and set Block AI bots to Do not block.

Common questions

Related articles

• Fix: Site can't be crawled (server errors, blocks and timeouts)
• Fix: 403 and 429 from the server, host firewall or ModSecurity (nginx, Apache, cPanel, Plesk, hPanel)
• Fix: WordPress security plugins block the crawl (Wordfence, Solid Security, Sucuri, LiteSpeed)
• Fix: Crawl finds few pages — redirects, robots.txt, login walls and JavaScript links
• Crawl a whole website
• An audit failed or did not finish

Still need help?

Ask in the chat inside the app, or send us a message on the contact page. You can also browse all help topics.