Start free trial
Site crawl, backlinks, research and AI visibility

Crawl a whole website

Crawl a client website to find broken pages, redirects and on-page issues. Up to 1,000 pages per crawl, one crawl at a time.

Updated Oct 24, 2026

A site crawl starts at the home page, follows every internal link and lists the issues it finds on each page.

Steps

  1. Open Site crawl and click Start crawl.
  2. Pick the Client website. If the site isn’t listed, add it as a client first (how).
  3. Choose Max pages: 500 for small sites or 1,000 for most client sites.
  4. Switch on Include subdomains to also crawl blog.site.com, shop.site.com and so on.
  5. Switch on Render JavaScript for React or Vue sites. It is slower.
  6. Under Repeat, choose Run once, Every week or Every month.
  7. Click Start crawl.

One crawl at a time

Each account runs one crawl at a time. If a crawl is already running, the new one shows Queued (#N) and starts automatically when the running one is done. The same site can’t be queued twice.

Read the results

Open the crawl from Site crawl to see the tabs Overview, Pages, Links, Resources, Issues, Compare and Change monitor.

  • Compare shows new, fixed and persisting issues between two crawls.
  • Change monitor tracks page changes; see Page change monitoring.
  • Crawl again starts a new crawl. While it runs, you see the previous crawl.
  • Export downloads the results.

Limits

  • Up to 1,000 pages per crawl on every plan.
  • Repeat crawls run 7 days (Every week) or 30 days (Every month) after the last crawl finished.
  • If a crawl runs longer than 45 minutes, the app stops following it with “The crawl took too long”.

Why a crawl finds only a few pages

The crawler starts at the client’s home page and follows the links it finds. If it finds only one page, or very few, check these in order:

  1. The site could not be loaded. If the only page shows a server error (5xx) or no status, the site blocked the crawler or was down. Some firewalls and bot protection block crawlers. Allow our crawler (the IP addresses and User-Agent are in Fix: Site can’t be crawled › What to allowlist), then click Crawl again.
  2. Links are built with JavaScript. Some sites add their menu and links only after JavaScript runs. Turn on Render JavaScript when you start the crawl. It renders the start page only, so see Fix: Crawl finds few pages for sites whose deeper pages also build links with JavaScript.
  3. robots.txt or noindex. Pages blocked by robots.txt (rules for SEOAuditSoftware or *) are skipped and not listed in the crawl. Overview › Indexability shows crawled pages that are noindex or canonicalised. See Fix: indexing.
  4. Redirects to another domain. If the home page redirects to another domain (for example from example.com to shop.example.net), the crawler may stop at the redirect. Add the final domain as the client’s website: in Clients, open the client › Settings tab (edit a client). For subdomains such as blog.example.com, turn on Include subdomains.
  5. The page limit. The crawl stops at the Max pages you chose (500 or 1,000).
  6. Pages behind a login. Pages that need a sign-in can’t be crawled (Fix: Site can’t be crawled › Login wall).

Crawl issues explained

The Issues tab groups problems by how much they matter: errors first, then warnings, then notices. Each issue lists the affected URLs.

  • Broken internal links (4xx) (error): Internal links point to pages that return 404. How to fix: Update each link to a live page, or add a 301 redirect from the old URL to the closest match. Full fix guide.
  • Server errors (5xx) (error): Pages that returned a 5xx error or no response at all (timeout, DNS, SSL or a blocked connection). How to fix: Open the Pages tab, filter by 5xx and look for “No response”, then follow the steps for that error type. Full fix guide.
  • Missing title tag (error): Pages without a <title> element. How to fix: Add a unique title of 50 to 60 characters that starts with the main keyword. Full fix guide.
  • Duplicate title tags (error): Several pages share the same title. How to fix: Write a unique title for every page so Google can tell them apart. Full fix guide.
  • Broken images (error): Images that return 404. How to fix: Upload the missing file again or remove the image from the page. Full fix guide.
  • Missing meta description (warning): Pages without a meta description. How to fix: Add a 140 to 155 character summary with the keyword and a clear reason to click. Full fix guide.
  • Title too long (over 60 characters) (warning): Titles that get cut off in search results. How to fix: Shorten the title and keep the keyword near the start. Full fix guide.
  • Images without alt text (warning): Images with an empty or missing alt attribute. How to fix: Describe each image in a few words. Skip alt text only for decorative images. Full fix guide.
  • Redirect chains (warning): Links that pass through two or more redirects. How to fix: Point links straight to the final URL and keep one 301 at most. Full fix guide.
  • Slow pages (over 3 s) (warning): Pages that took more than 3 seconds to load. How to fix: Compress large images, defer non-critical JavaScript and enable caching. Full fix guide.
  • Missing H1 (warning): Pages without a main heading. How to fix: Add one H1 that matches the topic of the page. Full fix guide.
  • Low word count (under 300) (notice): Thin pages with little text. How to fix: Add useful content: prices, FAQs, steps or local details. Full fix guide.
  • Multiple H1 tags (notice): Pages with more than one H1. How to fix: Keep one H1 and turn the others into H2 headings. Full fix guide.
  • Orphan pages (notice): Pages with no internal links pointing to them. How to fix: Link to these pages from the menu, related posts or service pages. Full fix guide.
  • URLs with parameters (notice): URLs with ? query strings that may create duplicates. How to fix: Add a canonical tag to the clean URL or block the parameter in robots.txt. Full fix guide.
  • Missing hreflang (notice): Language versions are not linked to each other. How to fix: Add hreflang tags if the site has more than one language. Ignore this if it does not. Full fix guide.

For a site that can’t be crawled at all (blocked, timeouts, server errors), start with Fix: Site can’t be crawled. All fix guides are in the Fix guides category of the help center.

The site health score goes up as you fix errors and warnings. Click Crawl again after fixing, then use Compare to see which issues are new, fixed or still there.

Common questions

How many pages can a crawl check?

Up to 1,000 pages per crawl, on every plan.

Can I run two crawls at once?

One crawl runs at a time per account. Other crawls wait in a queue and start automatically, one after another.

Why did my crawl find only one page (or very few)?

The crawler follows the links it finds on each page. It stops early when the site can't be loaded (it blocks bots or returns an error), when links are built with JavaScript, when robots.txt blocks pages, when the site redirects to another domain, or when the page limit is low. See Why a crawl finds only a few pages below, and the full guide Fix: Crawl finds few pages.

Do crawls count against a limit?

No. Crawls are not counted against a monthly limit, but they need an active plan or trial.

Still need help?

Ask in the chat inside the app, or send us a message on the contact page. You can also browse all help topics.