Skip to main content
All CollectionsGlobal Settings
Global crawler settings
Global crawler settings

Edit Sitebulb's default crawler settings

Updated over a month ago

Sitebulb has a range of 'global settings' which act as the default settings for every new audit you start. From the top navigation menu, choose the 'Settings' option.

Navigating to Global Settings

From here, the 'Crawler' tab allows you to adjust the following things the default crawler type so you can switch between HTML & Chrome crawling you can also select the default device type either Mobile or Desktop, the default User Agent and default Accept-Language, both of which are sent through on HTTP Header requests when Sitebulb crawls each URL.

Global crawler settings in Sitebulb

You can adjust these values using the drop-down menus, as above. Bear in mind that the values you select here may have an impact on how the server decides to return content to you. If you have trouble crawling a website, such as URLs returning 403 HTTP responses, it is always worth trying to change the user-agent and/or the accept-language values to see if the server treats these requests differently. You can do this within the Project settings.

Did this answer your question?