Monday, June 28, 2010

Comparing web application scanners, part 2

A new paper has been published by UCSB analysing the performance of various web application vulnerability scanners, which the authors say is "the largest evaluation of web application scanners in terms of the number of tested tools ... and the class of vulnerabilities analyzed".
The authors created their own test application containing a wide variety of vulnerabilities and crawling challenges, and carried out what appears to be a very detailed and rigorous analysis of each scanner's performance against this application.
Scanners were scored based on their ability to identify different types of vulnerabilities in different scanning modes. The overall scores, together with the prices of each scanner, were as follows:
ScannerScore  Price
Grendel-Scan  3Free
In addition to these core results, the authors also drew the following conclusions:
  • There are whole classes of vulnerabilities that cannot be detected by the state-of-the-art scanners, including weak passwords, broken access controls and logic flaws.
  • The crawling of modern web applications can be a serious challenge for today’s web vulnerability scanners, due to incomplete support for common client-side technologies and the complex stateful nature of today's applications.
  • There is no strong correlation between price and capability, as some of the free or very cost-effective scanners performed as well as scanners that cost thousands of dollars.
I must say, I completely agree with these conclusions. Firstly, Burp Scanner was designed with a clear awareness of the kinds of issues that scanners can reliably look for. It seeks to automate everything that can be reliably automated, giving you confidence in its output, and leaving you to focus on the aspects of the job that require human experience and intelligence to deliver. Secondly, devising a fully automated crawler that provides comprehensive coverage of today's applications, with their widely varied technologies and stateful designs, is a Herculean task. Even the best crawlers fall very far short of this, and claiming otherwise only gives false reassurance that this key part of application testing can be left to a machine. Burp Spider does provide crawling capabilities, both active and passive, but this feature is designed to be used in tandem with manual application mapping, and human sense-checking of the coverage achieved and the requests that need to be scanned for vulnerabilities.
I was, of course, pleased to see this recognition of Burp Scanner's capabilities, and the above comparison of scanners' performance versus price should make interesting reading for anyone who is deciding which products to spend their money on. Rest assured, I'll be going through the raw results from this survey in detail, and looking at ways to make Burp even more effective.


Anonymous said...

Congrats, burp is awesome and the best i have worked with.

Anonymous said...

Burp rocks... Simple as that.

Anonymous said...

It appears that these people did not work with the vendors as Suto did.

It looks like:
Appscan is roughly current
BURP is 1 major version back (18 months)
Cenzic is 2 major versions back (2008)
NTOSpider is 4 major versions back (2008)
WebInspect is 1 major version back (2008)

Given the rapid pace of change in this space, this is hardly an apples to apples comparison.

Anonymous said...

Was the textarea input issue resolved?

Anonymous said...

Not sure this is the appropriate place, but here is a feature request. With Intruder there is the capability to save an attack and then open saved attacks. For repeater it appears there is only the ability to "save item" or "save entire history," but not to reopen later. This would be great to have.

Anonymous said...

I am web app tester and I am using Burp every day,works like a charm..very fast and thx and congrats for the good work!

IceDragon said...

Just an FYI, guys: the PDF is no longer available (404)

A bit awkward, since this is linked to from the main pages. :P

Dafydd Stuttard said...

Thanks, we've removed the link since they're no longer hosting the paper.