Sitebulb is a desktop application, which means that in order to run, it needs to use the CPU and RAM of the machine it is installed on. It also means that when Sitebulb completes an audit, it writes data to the local hard drive (rather than to the cloud, as with cloud based crawlers like DeepCrawl or OnCrawl).
This means that, under certain conditions, Sitebulb can run out of resources. And when it does...bad things happen.
This is also true in general - trying to use a computer which has basically no resources left is an excruciating experience.
What can go wrong?
The three main resources that Sitebulb can exhaust are hard disk space, RAM and CPU, so we'll deal with each in turn:
Your computer has run out of hard disk space
This one is pretty straightforward. When crawling a website, Sitebulb is extracting a ton of data about the URLs, links and content on the site, and writing this data to disk so that it can be queried and analysed when viewing reports in Sitebulb.
Depending on the size of the site and the audit options you select, a single audit can take up anything from a few MB to a few GB. If you never delete any of your old audits, this data can start to pile up over time, to the point where you have little hard disk space remaining.
Of course this is not limited to Sitebulb data files - anything you store on your computer takes up hard disk space, such as images or video files.
Since Sitebulb requires at least some free space in order to write data to disk, if you hard drive ends up too full, it will not be ale to complete any audit it is doing.
So Sitebulb has a built in auto cut-off function, which will stops a running audit if your machine falls below 500MB free space. Your running audit will pause, and will display this red warning message, letting you know how much disk space you have left:
The solution is reasonably straightforward - go and delete some files! Initially, this might be as straightforward as simply emptying your recycle bin. In the long run, you will most likely need to clear out some more space though, so spend some time tracking down the programs/folders/files that take up most space on your machine, and delete anything you don't need.
If you find that Sitebulb's audit data itself is taking up lots of space, then consider following our guide on Managing file space for your Projects.
Your computer has run out of RAM
This one is less common, because your computer is often able to reallocate memory, but it can happen nonetheless, particularly on less powerful machines. The thing to watch out for is particularly memory-greedy programs (Chrome, I'm looking at you) or trying to run lots of intensive programs all at once.
When crawling in general, Sitebulb does not take up too much in terms of RAM, since it is committing data to the database rather than trying to hold it in random access memory, like some other tools do. However, if you have crawled a bigger site, Sitebulb can start to struggle a bit when producing the reports at the end of the audit. It is at this point that Sitebulb needs to take large chunks of data into RAM and manipulate it (in order to analyse), before again writing to disk.
If your machine starts to run low on RAM while Sitebulb is running, it will display a warning in the system tray;
This is the first stage of a 'three strike' warning process:
Trigger a warning when the memory is less than 512MB
Pause the audit when the memory is greater than 0 but less than 128MB for more than 6 minutes
Pause the audit when the memory gets to 0 bytes available
The reason we have these safeguards in place is because in this state you are teetering right on the edge of what your machine can feasibly handle, and pushing it any further would most likely cause your entire system to crash, losing any unsaved work in the process.
If you find yourself low on RAM, the best thing to do is pause your audit, close Sitebulb down then restart your machine. This clears off any memory that applications may have been holding onto, and basically gives you a fresh start. Then after the restart, only run Sitebulb and not all the other applications. Through experience we have found that some users never turn their machines off or restart them, and instead lean on the 'sleep' function all the time. This is one of the reasons why a proper restart is so effective.
The other option of course is to close down some of the applications that are hogging memory - even shutting them down and then starting them back up again can make a huge difference (including with Sitebulb). However our primary recommendation is a system restart.
What if this still doesn't work?
Some elements of the processing at the end of an audit - where Sitebulb produces reports and exports - are particularly memory intensive. If you are crawling a big site with tons of data, or if your machine is not particularly powerful, it is possible that Sitebulb will not actually be able to compete the audit at all.
This is because you are simply pushing you machine beyond its limitations. As a rough rule of thumb, you need 4GB RAM for every 500,000 URLs you audit - but this absolutely comes with the SEO-primed caveat of 'it depends' (on the size/type of the site, of the audit options you select, of the other processing running on your machine etc...).
Unfortunately what this means is that you may have crawled a massive site for several days and end up having to abandon the audit... an entirely unpleasant experience. If you end up in situation like this, please email us on [email protected], and we'll do our best to help.
One thing however is clear - this situation should only really be happening on particularly large audits, so you can self-help and prepare by reading our documentation on how to configure your machine for crawling large websites.
Your computer has maxed out CPU
This one is not a problem whereby Sitebulb will automatically pause (or eventually crash), but if you find your machine is using up lots of CPU you will notice that using it becomes slow and jerky, and your computer will sound like it's trying to take off.
Sitebulb can cause this, most commonly if you use lots of Chrome instances or switch on audit options like Performance, which take up a lot of processing power.
It is perfectly possible that you are comfortable with this situation;
You might be crawling a smaller site and know that the 'max CPU' situation is very temporary - once Sitebulb has completed it will drop back down.
You might be leaving Sitebulb to crawl overnight.
You might be looking for an excuse to avoid work ('Sorry boss, Sitebulb has locked my machine up, I'll pack it in for the day and go down the boozer instead').
If this is the case, there's nothing to do - just leave Sitebulb to run and it will be fine.
However, if the CPU situation is causing you issues, or meaning that you are not able to use your computer successfully for other tasks, the best course of action is to slow Sitebulb down by adjusting the speed settings in the Crawler Configuration. Lowering to 1 or 2 instances of Chrome can dramatically reduce the CPU it uses up.
Since you will most likely be doing this mid-crawl, you will need to pause the audit, update the settings, then resume:
