Carrying out a successful and efficient Technical SEO Audit starts with gathering the correct audit data.
This article is designed as a companion to Sitebulb's Technical SEO Auditing Fundamentals Course. We’ll walk you through each of the recommended crawl configuration settings you'll need to enable, in order to gather the necessary data points to complete this training course.
Note that you may have received specific instructions from your boss to change these configurations based on insights provided in the Setup Questions template (as filled out by or with your boss); make those alterations as needed.
Set up a sample crawl
For very small sites, you are welcome to skip this step and review Critical Errors (the first issue on the Audit Template, line #3) via a standard project crawl.
Create a new Sitebulb project
Name it (specifying Sample in the title to prevent any possible future confusion)
Set the domain homepage you are auditing on the URL
Under Audit Type, select Sample:
Click Save and Continue
On the next step with additional configuration options, click Start Now:
No custom configurations are necessary for the Sample Crawl. Once this project crawl is complete, proceed to the item check on line 3 of the template, and onto the Project Crawl.
Note: if there are setup errors and you can’t reach this Audit Data page to kick off the crawl, it’s likely due to an incorrect Start URL (learn about domain resolution errors) or a permissions issue (learn how to allowlist Sitebulb for crawling.)
Set up a standard crawl
The following steps outline the recommended default settings for a standard audit as a part of this Auditing Fundamentals course.
Create a new project
Name it
Set the domain homepage you are auditing on the URL
Under Crawler Type, select Chrome Crawler:
Click Save and Continue
Configure Audit Data
Here you get to choose what reports will be included in your final Audit.
For most cases, ensure that each of the following reports are selected (green/on):
Search engine optimization
Link analysis
Page resources
Performance & Mobile Friendly
Optionally, you can choose not to include Page Resources and Performance & Mobile-Friendly reports. This may be desirable when auditing very large sites (100k+ pages) as these reports will cause the crawl to take longer.
It will also mean, however, that the crawl won’t collect data for the following checks: image alt text, site performance (speed), and mobile-friendliness.
Configure Google Analytics
Connect Google Analytics and enable the ‘Extract and crawl any URLs found in Google Analytics’ feature.
Configure Google Search Console
Connect Google Search Console and enable the ‘Extract and crawl any URLs found in Google Search Console.
Configure Crawl Sources
By default, Sitebulb will only crawl the pages found through internal links; use the following instructions to ensure that you can easily audit the XML sitemap file directly in Sitebulb.
Tick XML Sitemaps to turn it on
Add any known xml sitemap URLs to the list:
You can choose to add URLs from other sources, including GA, GSC, and/or URLs from a list, especially if identifying Orphan Pages is important.
Configure Crawler Settings
For the most part you can ignore crawler settings, except for the following (optional) situations.
Set crawler speed limitations
If crawler speed limitations were requested by your boss/client, you can revise the defaults on Crawler Settings > Crawler, in Instances of Chrome, and the Max HTML URLs per Second. Simply change these fields to match the requested limits outlined on the Setup Questions document.
Get notified when the crawl is complete
If the site is large, it may be desirable to be automatically notified that the crawl is complete via email. If/when this is the case, on the Crawler Settings tab, scroll down to the Send Email When Finished Section and add the email address you wish to be alerted on:
Configure Subdomain Options
If specific subdomains should be included or excluded as a part of this audit (per Setup Questions insights from your boss), list those domains on the Subdomains Options tab.
Configure Robots Directives
Note: Learn more about auditing staging sites with Sitebulb - if this is applicable to the site you are reviewing.
Tick ‘Save disallowed URLs’:
Also consider checking the 'Crawl disallowed URLs' and 'Crawl links with nofollow directives' settings, giving Sitebulb full access to the content you may want to crawl and save.
Finalize settings and kickoff the crawl
Once you are happy with your crawl configurations, click Start Now in the bottom right corner:
With these changes you should be set. Go back to your SEO Audit template to take the next steps when you are ready to begin.