Setting up your Technical SEO Auditing crawl

Carrying out a successful and efficient Technical SEO Audit starts with gathering the correct audit data.

This article is designed as a companion to Sitebulb's Technical SEO Audit Checklist & Template. We will look at each template section and point you to the settings you'll need to enable in order to gather the data point you'll need to find useful insight and points of action.

Choosing the right crawler is essential in ensuring that all your page content is rendered and crawled. Sitebulb offers two crawler options:

Choose the Chrome crawler if you know your website renders content using JavaScript.

By default, Sitebulb will respect all robots directives, unless otherwise specified in your Robots Directives Audit Settings. As a result, disallowed directives will be respected, and disallowed pages will not be part of your audit by default.

If you wish to analyse how robots' directives impact the crawlability of your pages, you can choose to save disallowed URLs to your Audit by ticking ‘Save disallowed URLs’ under the Politeness tab in the Robots Directives settings:

Staging sites are likely to have directives in place to prevent the content from being crawled and indexed. Since Sitebulb respects robots directives by default, you will need to adjust your robots Politeness settings to ensure that Sitebulb can crawl successfully.

Navigate to the Robots Directives tab on your Audit Settings and find the Politeness section. Here, you can tick 'Is Staging Site' - this will remove the root path disallow from the robots.txt file, but instruct Sitebulb to respect the other robots directives.

Additionally, you can also choose to fully disable the 'Respect Robots Directives' setting, and even enable the 'Crawl disallowed URLs' and 'Crawl links with nofollow directives' settings, giving Sitebulb full access to the content you may want to crawl and save.

By default, Sitebulb will only crawl the pages found through internal links. You'll need to add your XML Sitemaps as a crawl source if you wish to analyse its content, find orphan pages, and pick up hreflang implemented through XML Sitemaps.

To crawl and analyse the content of your XML Sitemaps with Sitebulb, ensure that XML Sitemaps are selected as a Crawl Source in your Audit Settings.

By default, Sitebulb reports on all internal and external links found on your pages. When 'Crawler' is selected as the Crawl Source (default), Sitebulb discovers new internal pages by following internal links on every page it crawls.

External links are not crawled. By default, Sitebulb will only report on the HTTP status of all external links found on your pages.

To reduce the size of your audit, you can opt to exclude external links from your audit by disabling the External Link Analysis setting under Advanced Search Engine Optimisation Settings.

We have a whole article dedicated to how to find orphaned pages on your website. Here is a summary of the key steps to help you identify orphan pages in your Audit.

You will need to set up your audit to allow you to find as many of your Internal pages as possible. To find orphan pages, you need to compare crawl data with other URL sources.

To analyse alt text, you will first need to ensure that Page Resources (specifically, images) are part of your Audit.

Before starting your audit, navigate to Audit Settings > Audit Data and enable the Page Resources report - this will ensure that Sitebulb collects all page resources, including images, and can create reports based on this data.

Performance audits can be time and resource-intensive, as the crawler needs to collect and analyse every single page resource in order to report accurately on performance.

If you deal with bigger sites, you may want to consider setting up a separate Sitebulb Project specifically set to audit performance.

Since performance issues tend to be common across pages with the same templates/underlying HTML. As such, analysing a selection of key pages may be enough to find these bigger trends and opportunities for performance optimisation.

This may look like:

Once you have a clear idea of the scope of your performance Audit, go ahead and set up a new Project. Set your project name and start URL, then move on to the Audit Settings page, where you will need to:

You also have the option to adjust the options found under Advanced Settings. For more on what each of these options means, and how to adjust them, see our Performance Report documentation.

To instruct Sitebulb to gather and report on your structured data, you will need to enable the Structured Data report under Audit Settings > Audit Data:

If your website loads structured data via JavaScript, make sure to select the Chrome Crawler so Sitebulb can render and audit the markup correctly.

To gather International data, you’ll need to make sure that your International Report is enabled. To build your International Report, Sitebulb will crawl your entire site, including alternate URLs found in HTML, HTTP headers, or XML sitemaps, to find and audit international hreflang and HTML annotation.

Enable your International report under Audit Settings > Audit Data:

To review your website's security settings with Sitebulb, enable the Security report under Audit Settings > Audit Data:

Once you have the basics of your audit settings in place, you can go ahead and start your crawl. Alternatively, think about any other goals or requirements you may have for your audit, and follow the resources below to dig deeper into the audit settings: