Include & Exclude URLs settings

When you run a Sitebulb audit, the crawler starts at your homepage (or a URL you specify) and follows every link it finds, building up a picture of your website. But most websites have pages you don’t want to include in the audit - login pages, search results, faceted navigation URLs, staging environments, and more.

The Include & Exclude URLs settings give you precise control over what the crawler does and doesn’t visit. You can:

These settings are found in the “Include & Exclude URLs” section of the audit setup sidebar, under Crawler Settings. Inside, you will find eight sub-tabs. There is also a separate “Subdomain Options” tab in the sidebar.

If you simply want to crawl your entire website with no restrictions, you don’t need to change anything here. By default, Sitebulb crawls all internal pages it can find (subject to robots.txt rules, crawl depth limits, and page limits). The default settings work well for most standard audits.

You should configure these settings when:

Most of the tabs in this feature use a pattern language based on robots.txt directive syntax. If you’ve ever written a robots.txt file, you’ll recognise it immediately. If you haven’t, don’t worry - it’s straightforward once you understand a few rules.

This is important: patterns match against the path and query string portion of the URL only. They do not match against the protocol (https://) or the domain name (www.example.com).

For example, given the URL https://www.example.com/blog/article?page=2:

So the pattern you would write to match this URL is /blog/article* or /blog/* - you never include the domain.

Without a leading *: The pattern matches from the start of the URL path. Think of it as “the URL must begin with this.”

This matches: /blog/, /blog/my-post, /blog/2024/post This does NOT match: /en/blog/my-post (because the path starts with /en, not /blog)

With a leading *: The pattern matches anywhere in the URL path. Think of it as “the URL must contain this.”

This matches: /blog/my-post, /en/blog/my-post, /fr/blog/2024/post

With a trailing $: The pattern must match the exact end of the URL. Think of it as “the URL must end exactly here.”

This matches: /women This does NOT match: /women/coats, /women?page=2

If you write a pattern that starts with a question mark (to match query parameters), Sitebulb automatically treats it as if you wrote *? - meaning it will match the query string anywhere in the URL.

So these two patterns are equivalent:

Both match: /browse/home-garden?cardSize=small

Enter one pattern per line. Each pattern is evaluated independently - a URL only needs to match one pattern to be affected.

This would match any URL containing /blog/, /news/ at the start, or /archive/ anywhere in the path.
​

Note: Patterns are case-sensitive. The pattern /Blog/* will NOT match the URL /blog/my-post. Make sure your pattern’s letter casing matches the actual URLs on your website. If your site uses mixed casing, consider using the URL Rewriting feature to convert all URLs to lowercase before the patterns are applied.
​

Note: Do not use regular expressions (regex). Only the robots.txt wildcard syntax described above is supported. Writing regex patterns like ^/blog/.*$ will not work as expected.
​

The first sub-tab, and the one you’ll likely use most often. This is where you tell Sitebulb which internal pages or sections to skip during the crawl.

There are two separate controls on this tab: path exclusions and query parameter exclusions.

This is the main text area where you enter URL patterns for pages you want to exclude.

Any internal URL that matches one of your patterns will not be crawled. Sitebulb will still record that the URL exists (if it finds a link to it), but it won’t visit the page, download its content, or follow any links on it.

Below the path exclusion text area, you’ll find a separate section for handling query string parameters. Query strings are the parts of URLs that come after the ? - for example, in /products?sort=price&page=2, the query parameters are sort and page.

By default, Sitebulb does crawl URLs with query parameters. This means /products, /products?sort=price, and /products?sort=name are all treated as separate pages.

If you untick “Crawl Parameters”, Sitebulb will ignore all URLs that contain query parameters. Only the base URL (/products) would be crawled.

When you untick “Crawl Parameters”, a new text area appears: Safe Query String Parameters. This lets you make exceptions. Enter parameter names (one per line) that you still want Sitebulb to crawl.

For example, if your site uses page for pagination and you want to crawl paginated pages but nothing else:

Now Sitebulb will crawl /products?page=2 but skip /products?sort=price or /products?sessionid=abc123.

Note: If your site relies heavily on query parameters for navigation (/products?category=shoes), be careful with this setting. Turning off parameter crawling could cause Sitebulb to miss important pages.

This tab works identically to the path exclusion part of Internal URL Exclusions, but for external URLs - links pointing to other websites.

When Sitebulb finds links to external websites, it normally records them and checks their HTTP status (to find broken links, for example). External URL exclusions tell Sitebulb to skip checking certain external URL patterns entirely.

Note: Remember, these patterns match against the path and query string of external URLs, not the domain. If you want to exclude an entire external domain, use the External Domain Exclusions tab instead.

Inclusions are like a special case of exclusions. Instead of saying “crawl everything except these”, inclusions say “crawl only these and nothing else.”

When you add inclusion patterns, Sitebulb will only crawl internal URLs that match at least one of your patterns. Everything else is ignored. Think of it as putting a spotlight on just the pages you care about.

When you set inclusion patterns, a second text area appears: Links on Included Pages. This is an optional, more advanced feature.

Normally, when you use inclusion patterns, Sitebulb only crawls pages that match those patterns. But pages within your included section might link to pages outside it - for example, a blog post might link to a product page. Without this setting, those linked product pages would be ignored.

“Links on Included Pages” lets you define a secondary set of patterns. If a page matching your primary inclusion patterns links to a page matching these secondary patterns, that linked page will also be crawled - even though it doesn’t match the primary inclusion rules.

Primary inclusion pattern:

Links on Included Pages pattern:

Result: Sitebulb crawls all /blog/* pages. For any /products/* page linked from a blog post, Sitebulb will also crawl that product page.

The URL Seed List is conceptually different from the other tabs. Instead of patterns that match many URLs, you provide specific, complete URLs that you want Sitebulb to visit.

Seed URLs are added to the crawl queue alongside the URLs Sitebulb discovers by following links. The crawler will visit each seed URL and then follow any links it finds on those pages (subject to your other inclusion/exclusion rules).

Enter one full URL per line, including the protocol and domain:

https://www.example.com/blog/important-post https://www.example.com/products/featured-item https://www.example.com/landing-page-2024

Note: Seed URLs are still subject to your exclusion rules. If you add a seed URL that matches an exclusion pattern, it will still be excluded. Make sure your seed URLs don’t conflict with your exclusion rules.

URL Rewriting transforms URLs before the crawler processes them. This is different from exclusions - instead of skipping URLs, it changes them into a normalised form.

URL rewriting modifies URLs as Sitebulb discovers them, before any crawl decisions are made. This helps consolidate duplicate URLs that only differ in casing or unnecessary parameters.

There are two catch-all settings options you can enable:

When enabled, every internal URL is converted to lowercase before being processed.

​When you’d use it: Your site treats /Products/Shoes and /products/shoes as the same page, but the links on your site use inconsistent casing. Without this setting, Sitebulb would crawl both as separate pages.
​

Example:

Note: Only enable this if your server genuinely treats URLs as case-insensitive. On Linux servers, /Products/ and /products/ are often different pages. If in doubt, test by visiting both URLs in a browser.

When enabled, everything after the ? in a URL is stripped.
​

When you’d use it: Your site appends many different parameters that don’t change the page content, and you want a clean audit without parameter variations.
​

Example:

Instead of removing all parameters, you can specify individual parameter names to remove. Enter one per line.

When you’d use it: You want to keep meaningful parameters (like page for pagination) but remove tracking parameters.

Example - parameters to remove:

With these settings, the URL /blog/post?page=2&utm_source=google&fbclid=abc becomes /blog/post?page=2.

Live Test Tool

The URL Rewriting tab includes a live preview tool. Enter a URL and see immediately how your rewriting rules would transform it. Use this to verify your settings before starting the audit.

Note: URL rewriting happens early in the processing pipeline, before inclusion and exclusion patterns are evaluated. This means your patterns should match the rewritten form of URLs, not the original. For example, if you enable lowercase conversion, write your exclusion patterns in lowercase.

While External URL Exclusions (described above) let you exclude specific URL paths on external sites, External Domain Exclusions let you block entire domains.

Any external links pointing to excluded domains will be skipped entirely. Sitebulb won’t check their status, follow their redirects, or include them in your audit data.

There are two separate lists:

Enter one domain per line, without the protocol:

You can use *. at the beginning to match all subdomains of a domain:

This matches tracking.analytics-service.com, cdn.analytics-service.com, www.analytics-service.com, and any other subdomain.

Note: This tab uses plain domain names, NOT the robots.txt pattern syntax used in other tabs. Don’t add paths, protocols, or wildcards other than the *. prefix.

This tab controls which third-party resources (scripts, stylesheets, images, fonts, etc.) the crawler loads when visiting your pages. Blocking unnecessary resources can significantly speed up your audit.

When Sitebulb visits a page, it can load all the external resources that page references - JavaScript files, CSS stylesheets, fonts, images, tracking pixels, and more. Many of these resources are third-party services (analytics, advertising, chat widgets, consent banners) that aren’t relevant to your SEO audit.

Blocking these resources makes the crawl faster and reduces bandwidth usage without affecting the audit data you care about.

A dropdown at the top provides four preset levels:

For most audits, Third Party or Tracking Only is a good starting point. Use None only if you specifically need to analyse third-party resource loading (for page speed audits that depend on third-party scripts).

Below the preset dropdown, three sub-tabs let you fine-tune the blocking:

Enter domains (one per line) whose resources should be blocked. No protocol needed.

This sub-tab uses a different pattern syntax from the rest of the Include & Exclude URLs feature. Here, you use glob-style patterns that include the protocol:

The *:// matches any protocol (http or https). The * works as a wildcard for any characters, similar to the robots.txt syntax, but the full URL including protocol is used here.

The reverse of domain exclusions - domains entered here will never be blocked, even if they match a blocking category from the preset level. Use this for third-party services that are essential to how your pages render.

When you’d use it: You’ve set the blocking level to “Third Party” but your site uses a third-party JavaScript framework from a CDN that’s needed for the page to render properly. Add that CDN to the allowed list.

Note: The URL Path Exclusions sub-tab within Third Party Resources uses a different pattern format (full URL with protocol, glob-style) compared to the robots.txt style syntax used in the Internal/External URL Exclusions tabs. Don’t mix up the formats.

This tab works fundamentally differently from all the others. Instead of matching URLs, it matches HTML elements on the page using CSS selectors.

When Sitebulb parses a page, it extracts all the links it finds. Exclude Link Settings lets you tell Sitebulb to ignore links found inside specific HTML containers. The links are skipped entirely - the crawler won’t follow them or record them.

This operates at the HTML parsing stage, before any URL-level rules are applied.

Enter standard CSS selectors, one per line. If you’ve ever used CSS or browser developer tools, these will be familiar. If not, here are the most useful types:

E-commerce site with faceted navigation: Your product listing page has a sidebar with class facet-navigation that contains hundreds of filter links. Add:

All links inside any element with the class facet-navigation will be ignored.

Blog with a related posts widget: Your blog has a <div id="related-posts"> section at the bottom of each article. Add:

Multiple elements to exclude:

How to Find the Right CSS Selector

Note: These are standard CSS selectors, not URL patterns. Don’t enter URLs or robots.txt patterns here.

Note: Links excluded this way are completely invisible to the crawl. They won’t appear in any reports and the crawler won’t follow them. Make sure you’re not accidentally excluding important navigation.

This is a separate top-level tab in the audit setup sidebar (not a sub-tab of Include & Exclude URLs), but it’s closely related to URL inclusion and exclusion behaviour.

When Sitebulb encounters links to subdomains of your primary domain (blog.example.com when auditing www.example.com), this setting controls how those subdomains are treated.

Depending on your chosen option, additional text areas appear where you can list specific subdomains:

When “Audit and Report” is selected: You can list specific subdomains to exclude from the audit. All other subdomains will be audited.

When “Exclude All” is selected: You can list specific subdomains to include in the audit. All other subdomains will be excluded.

Enter subdomain names (without the main domain), one per line:

Note: When both inclusion and exclusion subdomain lists are in play, inclusions take priority over exclusions. If a subdomain appears in both lists, it will be included.

When you configure multiple rules across different tabs, Sitebulb evaluates them in a specific order. Understanding this order helps you avoid unexpected behaviour.

For every URL the crawler encounters, Sitebulb follows this decision process:

If a URL matches both an exclusion pattern and an inclusion pattern, the exclusion wins. Exclusions are evaluated first, and an excluded URL will not be crawled regardless of any inclusion rules.

To avoid this, make sure your exclusion and inclusion patterns don’t overlap in unintended ways.

The one scenario where an excluded URL can be “rescued” is through secondary inclusion (“Links on Included Pages”). If a URL was excluded but it matches a secondary inclusion pattern AND it was linked from a primary-included page, the secondary inclusion can override the exclusion.

However, this only works one level deep and is a niche scenario. For most users, the simpler rule is: exclusions always win.

The Exclude Link Settings (CSS selectors) operate at the HTML parsing stage - before URLs even enter the scheduling pipeline. If a link is inside an excluded CSS selector, the URL is never extracted from the page in the first place, so none of the URL-based rules even see it.

Because URL rewriting happens before pattern matching, your patterns need to match the rewritten URLs, not the originals.

The mistake: Writing products/* instead of /products/*.

Why it’s wrong: All URL paths start with /. Without it, the pattern has no way to match the beginning of a path. The crawler will never find a URL whose path starts with products (no slash).

The fix: Always start your patterns with / (to match from the beginning of the path) or * (to match anywhere in the path).

The mistake: Writing /blog/* when your site’s URLs use /Blog/my-post.

Why it’s wrong: Pattern matching is case-sensitive. /blog/* and /Blog/* are completely different patterns.

The fix: Check the actual URLs on your site (look in the browser address bar). Match the exact casing. If your site uses inconsistent casing, enable “Convert all URLs to lowercase” in the URL Rewriting tab and write all your patterns in lowercase.

The mistake: Adding an inclusion pattern for /resources/* when your start URL (homepage) doesn’t link to any /resources/ page.

Why it’s wrong: Sitebulb follows links from your start URL. If no link leads to a page matching your inclusion pattern, the crawler discovers nothing.

The fix: Either change your start URL to a page within the included section, or add seed URLs (in the URL Seed List tab) that match your inclusion patterns.

The mistake: Adding an exclusion for */archive/* and then wondering why /blog/archive/2023/post isn’t crawled, even though /blog/* is in your inclusions.

Why it’s wrong: Exclusion rules are always evaluated first and always win.

The fix: Review both your exclusion and inclusion lists together. If you want certain URLs to be crawled, make sure they don’t match any exclusion pattern. Remove or narrow the conflicting exclusion.

The mistake: Entering a full URL with protocol in the Internal URL Exclusions tab, or entering a path-only pattern in the Third Party Resources URL Path Exclusions.

Why it’s wrong: Different tabs use different formats:

The fix: Refer to the specific tab’s instructions and use the correct format for that tab.

The mistake: Using /p* to exclude /products/ but accidentally also excluding /pages/, /privacy-policy, and /pricing.

Why it’s wrong: * is greedy and matches everything. A short prefix pattern like /p* will match far more URLs than you intend.

The fix: Be as specific as possible. Use /products/* instead of /p*. Test your patterns mentally against URLs you do want to crawl to make sure they won’t be accidentally caught.

The mistake: Setting up secondary inclusion expecting it to cascade through multiple levels of linked pages.

Why it’s wrong: If Page A (primary included) links to Page B (secondary included), and Page B links to Page C, Page C is NOT automatically included. Only the direct links from primary-included pages are rescued by secondary inclusion.

The fix: If you need deeper cascading, expand your primary inclusion patterns instead of relying on secondary inclusion.

When you don’t configure any inclusion or exclusion rules: