Website Crawler for SEO Audits
Crawl a client’s whole website, up to 1,000 pages, and find broken links, server errors, missing titles, slow pages and 12 more issue types on every page. Each issue comes with the affected URLs and a fix guide. Repeat the crawl every week or month and compare crawls to see what was fixed and what is new.
No credit card needed. Crawls don’t count against a monthly limit.
- 1,000pages per crawl
- 16issue types with fixes
- 7result tabs per crawl
- CSVexport of pages and issues
How to crawl a website for SEO issues in 4 steps
The crawler starts at the client’s home page, follows every internal link it finds and checks each page it reaches.
Start a crawl
Open Site crawl, click Start crawl and pick the client website.
Choose the options
Max pages (500 or 1,000), include subdomains, render JavaScript, and run once, every week or every month.
Let it run
One crawl runs at a time per account. Others wait in the queue and start automatically, one after another.
Fix and compare
Work through the issues, click Crawl again and use Compare to see new, fixed and persisting issues.
Everything the site crawler checks and shows
A full technical SEO crawl of each client website, with real counts from the crawled pages and a fix for every issue.
Up to 1,000 pages per crawl
Crawl 500 pages for small sites or 1,000 for most client sites. Add subdomains such as blog. and shop., and render JavaScript for React or Vue sites.
One crawl at a time, with a queue
Start crawls for several clients at once. One runs, the rest show their place in the queue and start on their own when it finishes.
Repeat crawls every week or month
Set a crawl to repeat. The next one starts 7 days (every week) or 30 days (every month) after the last crawl finished.
16 issue types, sorted by severity
Errors first, then warnings, then notices. Each issue lists every affected URL and how to fix it, with a link to the full fix guide.
See all 16 issuesHTTP status and indexability
See how many pages return 2xx, 3xx, 4xx or 5xx or no response at all, and how many are indexable, noindex or canonicalised.
Compare between crawls
Pick any two crawls of the same site and see which issues are new, which are fixed and which persist, with the health score before and after.
Pages, links and resources
Every page with status, title, words, click depth, indexability and load time. Every link with type, status, anchor text and where it was found. Every image, CSS and JavaScript file, with large files flagged.
CSV export
Export every crawled page, the full issue list, or the affected URLs of one issue as CSV, ready for a spreadsheet or the client’s developer.
Clear reasons when a site blocks the crawler
If the start page can’t be crawled, the app says why (bot protection, firewall, rate limit, timeout, DNS or SSL), what to do, and the crawler identity the site should allowlist.
What to allowlistWhy crawl the whole website, not one page?
A page audit checks one page in depth: 179 on-page checks, or a score for one target keyword. A site crawl checks every page it can reach for the problems that hide across a site: links to deleted pages, the same title on dozens of pages, redirect chains, orphan pages and slow templates.
Use both. Crawl the site to find where the problems are, then open any crawled page and click Run page audit for the full on-page report. Website audit tool
What a site crawl finds that a page audit can’t
- Internal links that point to 404 pages, and where they are
- Pages that share the same title tag
- Redirect chains with two or more hops
- Orphan pages with no internal links to them
- Slow pages and large images across the site
- Noindex and canonicalised pages you didn’t expect
- Server errors and pages that don’t answer at all
The 16 issues the website crawler reports
Every issue lists the affected URLs and how to fix it. The site health score goes up as you fix errors and warnings.
| Issue | Severity | How to fix |
|---|---|---|
| Broken internal links (4xx) | Error | Point each link to a live page or 301 redirect the old URL. Fix guide |
| Server errors (5xx) | Error | Find the error type (5xx or no response) in the Pages tab and fix it. Fix guide |
| Missing title tag | Error | Add a unique title of 50 to 60 characters. Fix guide |
| Duplicate title tags | Error | Write a unique title for every page. Fix guide |
| Broken images | Error | Upload the missing file again or remove the image. Fix guide |
| Missing meta description | Warning | Add a 140 to 155 character summary with a reason to click. Fix guide |
| Title too long (over 60 characters) | Warning | Shorten it and keep the keyword near the start. Fix guide |
| Images without alt text | Warning | Describe each image in a few words. Fix guide |
| Redirect chains | Warning | Link straight to the final URL, one 301 at most. Fix guide |
| Slow pages (over 3 s) | Warning | Compress images, defer JavaScript, enable caching. Fix guide |
| Missing H1 | Warning | Add one H1 that matches the topic of the page. Fix guide |
| Low word count (under 300) | Notice | Add useful content: prices, FAQs, steps or local details. Fix guide |
| Multiple H1 tags | Notice | Keep one H1 and turn the others into H2. Fix guide |
| Orphan pages | Notice | Link to them from the menu, related posts or service pages. Fix guide |
| URLs with parameters | Notice | Canonical to the clean URL or block the parameter. Fix guide |
| Missing hreflang | Notice | Add hreflang if the site has more than one language. Fix guide |
What you get for client reporting
Every crawl gives you numbers a client understands: the site health score, how many issues you fixed since the last crawl and what is new. Page change monitoring adds a log of every edit to the client’s key pages.
- Compare: new, fixed and persisting issues, and the health score before and after
- Pages CSV: URL, status, title, words, depth, indexability, load time and internal links in
- Issues CSV: every issue with its severity, number of URLs and how to fix it
- Affected URLs of one issue as CSV, ready to hand to the developer
- Change history of monitored pages with the old and new value
Crawl results are shared as CSV exports from the app. Scheduled white label client reports add the SEO audit summary, rankings, Search Console and Google Analytics 4 data.
Who uses the site crawler
SEO agencies
Crawl every new client site on day one, set a monthly repeat crawl and show progress with Compare in each monthly update.
Freelancers
Run a technical review of a prospect’s site before the sales call, and hand the issue list to the developer as CSV.
Web designers and developers
Crawl a relaunch or redesign to catch broken links, redirect chains and missing titles before the client does.
Fix guides and help for site crawl issues
Each fix guide explains how the issue is detected, why it matters and how to fix it, from a basic fix to an expert fix, with steps for WordPress, Shopify and other platforms.
Site crawl on every plan
Site crawls and page change monitoring are part of the software, next to audits, client reports and rank tracking. Try everything free for 14 days, with no credit card.
Website crawler FAQs
How many pages can the website crawler check?
Up to 1,000 pages per crawl. When you start a crawl, choose 500 pages for small sites or 1,000 for most client sites. The crawl stops when it reaches that number.
Can I run more than one crawl at a time?
One crawl runs at a time per account. If you start more, they show as Queued with their place in line and start automatically, one after another. The same site can’t be queued twice.
Does the crawler render JavaScript?
Yes, when you switch on Render JavaScript. Use it for sites that add their menu and links with JavaScript, such as React or Vue sites. It renders the start page and is slower than a normal crawl.
Does the crawler follow robots.txt and read the XML sitemap?
It follows robots.txt, using the rules for the SEOAuditSoftware user agent or the * group. It doesn’t read the XML sitemap: it starts at the home page and follows the links it finds, so pages that are only in the sitemap and not linked anywhere aren’t found.
Why did my crawl find only one page?
Usually the site blocked the crawler or was down, the links are built with JavaScript, robots.txt blocks the other pages, or the home page redirects to another domain. The app shows the reason on the crawl, and the guide Fix: crawl finds few pages walks through each cause.
What should a site allowlist so the crawler can get in?
Allowlist the crawler’s User-Agent, which contains SEOAuditSoftware, and our IP addresses in the firewall, bot protection or security plugin. The current IP addresses and step-by-step instructions are in Fix: site can’t be crawled.
Do crawls count against a monthly limit?
No. Crawls are not counted against a monthly limit, but they need an active plan or trial. See plans and pricing.
Crawl your first client site free
Start a 14-day free trial with no credit card and see every issue on the site in one crawl.