Occasionally, you'll find that your Sitebulb Audits do not return the expected results. When crawling Shopify websites, this is often due to the fact that Sitebulb ran into a 429 'Too Many Requests' error, as Shopify employs rate-limiting measures.
Sitebulb stops crawling once it receives a 429 response from the server, since no data can be collected for the URLs returning this HTTP status. To overcome these errors and successfully crawl your website, you will need to whitelist Sitebulb or otherwise authenticate the crawler on the website server.
In the case of Shopify websites, that means providing an HTTP message signature in the HTTP headers of each request, allowing Shopify to verify that the request is coming from an authorized crawler.
Follow the steps below to set up the necessary authorization to crawl Shopify websites successfully.
Create your HTTP message signatures in Shopify
Follow the steps provided in the Shopify documentation to create your signatures:
Online Store > Preferences in Shopify admin
In the Signatures section, click Create signature.
In the Name field, enter a descriptive name for your signature.
In the Domain field, select the domain that you want to use this signature for.
In the Valid for section, select an expiration period for your signature.
Click Create.
Each signature consists of three HTTP headers, which you'll need to add to your Project settings using Sitebulb's Custom Headers feature.
Add Custom Headers to your Audit Settings
Once you have created the signatures, add the resulting HTTP Headers to your audit settings to ensure they are part of the HTTP header requests sent to your website server by Sitebulb.
Navigate to Crawler Settings.
Scroll to find the Custom Headers settings area and add your HTTP Headers.
Each signature consists of three HTTP headers. Add each to a new line:
You can now go ahead and start your crawl. Sitebulb will send the custom HTTP headers with every request, ensuring that your Shopify website can be crawled successfully.