Looking for our research? We've moved it to a dedicated page

Barriers to automation 1 - vulnerability scanners

Dafydd Stuttard | 22 May 2007 at 23:24 UTC
automation Pen Testing scanners

When you are attacking a web application, automation is your friend. Not only if you are lazy, but also because automation can make your attacks faster, more reliable and more effective. This is the first in a series of posts exploring ways of using automation in web application testing, and the limitations that exist on its effective use.

Web application vulnerability scanners seek to automate many of the tasks involved in attacking an application, from initial mapping through to probing for common vulnerabilities. I've used several of the available products, and they do a decent job of carrying out these tasks. But even the best current scanners do not detect all or even a majority of the vulnerabilities in a typical application.

Scanners are effective at detecting vulnerabilities which have a standard signature. The scanner works by sending a crafted request designed to trigger that signature if the vulnerability is present. It then reviews the response to determine whether it contains the signature; if so, the scanner reports the vulnerability.

Plenty of important bugs can be detected in this way with a degree of reliability, for example:

However, not every vulnerability in the above categories will be detected using standard signature-based checks. Further, there are many categories of vulnerability which cannot be probed for in this manner, and which today's scanners are not able to detect in an automated way. These limitations arise from various inherent barriers to automation that affect computers in general:

The barriers to automation described above will only really be addressed through the incorporation of full artificial intelligence capabilities into vulnerability scanners. In the meantime, these barriers entail that many important categories of vulnerability cannot be reliably detected by today's automated scanners, for example:

Further, even amongst the types of vulnerability that scanners are able to detect, such as SQL injection and XSS, there are many instances of these flaws which scanners do not identify, because they can only be exploited by modifying several parameters at once, or by using crafted input to beat defective input validation, or by exploiting several different pieces of observed behaviour which together make the application vulnerable to attack.

Current scanners implement manual workarounds to help them identify some of the vulnerabilities that are inherently problematic for them. For example, some scanners can be configured with multiple sets of credentials for accounts with different types of access. They will attempt to access all of the discovered functionality within each user context, to identify what segregation of access is actually implemented. However, this still requires an intelligent user to review the results, and determine whether the actual segregation of access is in line with the application's requirements for access control.

Automated scanners are often useful as a means of discovering some of an application's vulnerabilities quickly, and of obtaining an overview of its content and functionality. However, no serious security tester should be willing to rely solely upon the results of a scanner. Many of the defects which scanners are inherently unable to detect can be classified as "low hanging fruit" - that is, capable of being discovered and exploited by a human attacker with modest skills. Receiving a clean bill of health from today's scanners provides no assurance that an application does not contain many important vulnerabilities in this category.