You run a website crawl and get a list of errors. But did the tool check your main service pages? Did it stop early? Was it looking at the right part of your website?
Those questions come before the fix list.
A useful Screaming Frog crawl setup gives you results you can explain. This guide helps you choose a starting point, understand the free version, check coverage, and turn a finding into a clear task.
Quick answer: For a first crawl, choose Spider mode, enter your live homepage URL, and start with the default setup. Watch the early results. Before making changes to your website, check that the crawl reached your important pages. Advanced settings, JavaScript rendering, and saving crawl sessions require a paid licence.
This is a guide based on current product documentation. The worked example is fictional, not a report from a client website.
What should your first crawl answer?
Choose one question you want the crawl to help answer.
For a local business, that might be: “Can visitors reach each main service page and then request a quote?”
For a small online shop, it might be: “Do our main product categories lead to working product pages?”
Write down the pages involved before opening the tool. A short list makes it easier to notice a missing section later.
| Page type | Why it belongs in your check |
|---|---|
| Homepage | Often the starting point for visitors |
| Main service or category pages | Explain what the business offers |
| Contact or booking page | Helps a visitor take the next step |
| About page | Gives readers business context |
| A few useful articles | Shows whether supporting content connects to services |
Do not judge success by the number of warnings found. Judge whether you have enough reliable evidence to make a useful decision.
For a broader review beyond crawl setup, use our technical SEO checklist.
Free or paid: what can you use?
According to Screaming Frog’s getting started guide, the free version crawls up to 500 URLs at a time. A licence removes that restriction and unlocks configuration and advanced features. Saving and reopening crawl sessions also requires a licence.
| Your task | Starting choice |
|---|---|
| Try a basic crawl of a small website | Free version |
| Review a site that exceeds the free crawl allowance | Licensed version |
| Change advanced crawl settings | Licensed version |
| Render JavaScript during the crawl | Licensed version |
| Save and reopen the full crawl session | Licensed version |
The limit is expressed in URLs. Do not treat “500 URLs” as a promise that every page on a website with 500 articles will be covered.
For example, a small business may have service pages, blog archives, images, and several versions of the same address. Your article count is not a dependable estimate of crawl size.
Start with the question you need answered. Upgrade when the task needs it, rather than assuming a paid tool will make every result more useful.
Step 1: Write a simple crawl brief
A crawl brief can be five lines in a document. It records what you meant to test.
Here is a fictional example for a cleaning business:
| Item | Example brief |
|---|---|
| Website | https://example.com/ |
| Goal | Check service pages and the route to the quote form |
| Included area | Public business website |
| Important pages | Homepage, three services, contact, about |
| Review needed | Missing pages, broken destinations, unclear next steps |
Also record the date and whether you are using the free or licensed version. If a developer is changing navigation that day, note it.
Without this context, two reports can appear to disagree when they checked different things.
You can keep these notes beside your findings in an SEO audit template.
Step 2: Choose the right starting URL and mode
Open your homepage in a normal browser. Copy the final address after any redirect. Check the spelling and whether the site uses www.
In Screaming Frog, select Mode > Spider, paste the address into the URL field, and click Start. Spider mode follows discovered links. Mode > List checks supplied URLs instead. Use List mode when you have a specific set to test; it does not by itself prove those pages are reachable through navigation.
For a general first review, use the homepage as your starting point. Starting from a blog folder answers a narrower question and can leave important business pages outside your intended scope.
Keep the first run simple. Avoid importing a colleague’s saved configuration unless you understand what it includes and excludes.
A direct URL check answers a different question
Imagine that a quote page opens when you paste its address into a browser. That tells you the address can load in that browser session.
It does not tell you whether a new visitor can find it from the homepage.
Keep both questions in your review: “Does the destination work?” and “Can a visitor reach it?”
Step 3: Review configuration only where the task needs it
Many configuration controls require a licence. If you use the free version, work with the default crawl and record its limits.
The official Screaming Frog configuration guide explains the controls. Useful areas include:
| Control | What it changes |
|---|---|
| Spider > Limits | Total URLs and how far the crawl can follow links |
| Include and Exclude | Which URL patterns are within the crawl |
| Speed | Request rate and concurrent activity |
| Robots.txt settings | How the crawler handles crawl rules |
| Spider > Rendering | Whether JavaScript is rendered |
At Configuration > Speed, the Max URI/s control limits requests per second. The guide gives 1 URL per second as an example of throttling, not a universal target. Agree on a suitable rate with the person managing the site.
Keep robots rules respected for the initial baseline. If you later run an authorized diagnostic with different rules, label it separately.
Use a reason for every restriction
Before adding an exclusion, write a sentence explaining it.
“Exclude account pages because this review covers public service content” is a clear reason.
“Exclude everything with a question mark because the report looks too large” needs more thought. You could remove addresses that matter to the investigation.
The same principle applies to a depth limit. A small limit can create a tidy report while leaving out pages you meant to inspect.
If you deliberately run a sample, call it a sample. A smaller report is useful when its boundaries are clear.
Step 4: Decide whether JavaScript rendering is needed
Some websites add important text or links after the first HTML response loads. A text-only crawl may therefore show less than you see in the browser.
Screaming Frog’s JavaScript tutorial explains how to compare the original and rendered page. With a licence, use Configuration > Spider > Rendering to select JavaScript when the site needs it. Rendering takes more resources than a text-only crawl.
A practical check is to choose one service page and one page that links to it. Compare their main content and links in the original HTML and the rendered view. If important elements appear only after JavaScript runs, investigate that difference.
Do not switch modes just because the first crawl looks short. A short crawl can have other causes.
Likewise, a successful rendered crawl does not mean every visitor interaction works. A quote form may still fail after someone fills it in.
Step 5: Watch the first results before leaving it running
Treat the beginning of the crawl as a short quality check.
Ask whether the addresses look familiar. Are they on the expected domain? Are service pages appearing? Is the report filling with repeated variations that were not part of your brief?
If the live site becomes slow or the crawler receives repeated errors, pause and investigate with the site manager. Do not keep increasing the crawl speed to finish sooner.
Record what you observe before changing a setting. Otherwise, you may lose the clue that explains the problem.
For example, “The site returned errors after the crawl began” is more useful than “Screaming Frog found a bad website.” The first statement describes evidence. The second jumps to a conclusion.
Step 6: Check coverage before reading the full issue list
Compare the results with the important-page list from your brief.
Screaming Frog’s missing-page guide recommends tracing the link path to a missing page. Check whether the page that should link to it was found, then inspect that link. Configuration, connection problems, and links that depend on JavaScript can affect discovery.
| What you notice | First question to investigate |
|---|---|
| An entire section is absent | Was it inside the crawl scope? |
| One service page is missing | Which page should link to it? |
| Only a few addresses appear | Did the crawler receive the expected content and links? |
| Results stop at the free limit | Was this an incomplete crawl? |
| Several important URLs show errors | Are those responses repeatable? |
Do not label a page an orphan simply because it is absent from one run. First rule out the limits of your own test.
Worked example: the missing services were outside the crawl scope
Imagine a cleaning company with three service pages and a quote form. The owner starts a crawl from the blog folder and restricts it to that folder.
The report contains articles but none of the service pages. It would be easy to conclude that the services have poor internal links.
Instead, compare the crawl brief with the setup.
| Check | Fictional finding | Decision |
|---|---|---|
| Starting point | Blog folder | Does not match the whole-site brief |
| Included area | Blog only | Service pages were outside the test |
| Browser navigation | Homepage links to the services | Investigate with matching crawl scope |
| New baseline | Homepage start, intended public site included | Recheck coverage |
In this example, the next crawl finds the service pages. No navigation change was needed to resolve the apparent gap.
Now imagine that one service page contains an old quote link. That is a separate finding worth checking in the browser.
The task should name the source page, the old destination, and the correct destination. After the change, test the route again and submit a test enquiry with the site owner’s approval.
This example shows why setup comes first. Fixing a real customer problem is useful. Changing a working website to satisfy an incomplete report wastes time.
Step 7: Keep only the evidence needed for each task
You can export tab data using the Export button. A spreadsheet export is different from saving the full crawl session.
For each finding, record what someone needs to reproduce it and confirm the fix:
| Field | Example task entry |
|---|---|
| Source page | Office cleaning service page |
| Observation | Quote button points to an old destination |
| User impact | Visitor cannot complete the intended enquiry route |
| Evidence | Source URL, destination URL, browser check |
| Owner | Website editor |
| Completion check | Correct link works and test enquiry arrives |
Avoid sending a developer a large export with “please fix SEO” as the only instruction.
A useful task has a clear finish line. “Check the contact route from the service page and confirm the enquiry arrives” is easier to act on than “improve conversions.”
Our worked SEO audit report shows how to present findings with business context.
Step 8: Make the next crawl comparable
Before comparing two runs, ask whether they tested the same thing.
Keep a note of the starting URL, included areas, rendering mode, restrictions, and whether each crawl completed. If one used a small sample and the other checked the whole public site, their totals will not be a fair comparison.
When you change a setting to investigate a problem, give the diagnostic run a clear name. Keep the baseline separate.
For example:
- Baseline: public website from homepage.
- Diagnostic: JavaScript rendering on selected templates.
- Follow-up: repeat baseline after the approved fix.
The goal is to explain why the evidence changed. A lower issue count alone does not show that customers have a better experience.
How does crawl setup relate to SEO, AEO, and GEO?
A crawl can support a technical review. It does not measure every part of search performance or tell you whether an AI answer will recommend your business.
Keep these outcomes separate in your reporting:
- Technical finding: what your crawl and manual checks observed.
- Search outcome: what happened to the new page’s impressions, clicks, and enquiries.
- AI visibility observation: whether a named answer service cited your content in a recorded test.
Do not turn “our crawler reached the page” into “LLMs will cite it.” Those are different claims and need different evidence.
For the content itself, explain the service clearly, answer real questions, and support factual claims with sources readers can inspect. A crawl report cannot replace that editorial work.
Common questions
What is the best Screaming Frog configuration for a small site?
Start with a setup that matches the question you want answered. For a basic first review, use the live homepage and default Spider mode. Check coverage before changing advanced settings.
Why does Screaming Frog stop at 500 URLs?
That is the free version’s crawl limit. Record the crawl as partial if it stops before your intended review is complete.
Does a missing page mean Google cannot find it?
No. Investigate your crawl’s scope, access, and link discovery first. One tool run is not a complete record of a search engine’s activity.
Should I enable JavaScript for every crawl?
Use it when the website or investigation needs rendered content. Test representative pages and understand the extra processing involved.
Does a clean crawl mean my contact form works?
No. Check the form as a visitor and confirm that a test enquiry reaches the intended person. Keep that result separate from the crawl findings.
Need help turning a crawl into useful fixes?
Start with your important pages and a clear question. Check the setup, confirm coverage, and give each real issue a practical next step.
Explore Worsoo’s SEO services and share your website, main services, and the problem you want to solve. We can discuss an audit scope that fits your business.