Friday, 25 March 2011
There are probably a few bugs to flush out, and I'll hopefully be adding a few more things during the beta period. Any feedback and bug reports would be much appreciated. Please email these directly so I can get back to you for more details if required.
Due to the phenomenal time sink that is the WAHH rewrite, this release will be in beta for at least a month. For users of the free edition who cannot wait that long, there is an easy solution.
The functionality needed to let Burp automatically handle a wide variety of session handling challenges is necessarily complex, and often requires a lot of careful configuration. The best way to illustrate the power of the new features, and show how the configuration works in practice, is via an example.
Let's look at an application function which can only be accessed within an authenticated session, and employs a further token to defend against CSRF attacks. You want to test this function for various input-based vulnerabilities like XSS and SQL injection. Previously, performing automated (and some manual) testing of this function would have faced two challenges: (a) ensuring that the session being used remained valid; and (b) obtaining a valid token to use in each request. The new functionality can take care of both these challenges.
To do this, we're going to define some session handling rules. These rules will be applied to each request that is made to the function we are testing by the Intruder, Scanner and Repeater tools:
Check whether the current session is valid, by requesting the user's landing page in the application, and inspecting the response to confirm that the user is still logged in.
If the user is not logged in, log them back in to obtain a valid session.
Request the page containing the form whose submission we are going to test. This form contains the anti-CSRF token that we need, within a hidden field.
Update the request to the function we are testing with the value of the anti-CSRF token.
In most situations, we need to make use of Burp's own session handling cookie jar, so the first rule we define tells Burp to add cookies from the cookie jar to every request. This is, in fact, the default rule for the Scanner and Spider tools, so we'll just modify the default rule to apply to the Intruder and Repeater tools as well. This rule performs a single action, shown below:
The rule's scope is defined to include the relevant tools, and apply to all URLs:
Next, we need to check that the user's current session on the target application is valid. Assuming we want to apply this rule to all requests within the target application, we can define it to be in-scope for the whole of the application's domain:
We then add a suitable description and add an action of the type "check session is valid":
This opens the editor for this type of action, which contains a lot of configuration options:
The first set of options determines which request Burp uses to validate the current session. The options are:
Issue the actual request that is currently being processed. This option is ideal if the application always responds to out-of-session requests with a common response signature, such as a redirection to the login.
Run a macro, to make one or more other requests. This option is ideal if, to identify whether the session is valid, you need to request a standard item, such as the user's home page. It is also the best option if you need to apply further rules to modify the request currently being processed - for example (as in the present case) to update an anti-CSRF token in the request. If the option to run a macro is selected, you have a further option whether to do this every for every request, or only every N requests. If the application is aggressive in terminating sessions in response to unexpected input, it is recommended that you validate the session every time; otherwise, you can speed things up by only validating the session periodically.
For the current example, we are going to run a macro to fetch the user's landing page in the application, to check that their session is valid. To do this, we need to define our macro, by clicking on the "new" button in the previous screenshot. This opens the macro recorder, enabling us to select the request(s) that we wish to include in the macro. In the present case, we only need to select the GET request for the user's landing page:
The second set of options in the "check session is valid" action controls how Burp inspects the (final) response from the macro to determine whether the session is valid. Various options are available, and the configuration we need in the present case is shown below:
The final set of options for this action determines how Burp will behave depending on whether the current session is valid:
You can tell Burp not to perform any further actions for this request if the session is valid. Using this option lets you define subsequent, separate actions to recover a valid session. This option is mandatory if the request itself has already been issued in order to determine whether the session is valid.
You can tell Burp to perform a sub-action if the session is invalid, and then continue to process subsequent actions. This is useful when you need to define subsequent actions in any case, following the session validity check, for example to run a macro to obtain a request token or modify the application's state.
In the present example, we need to use the second option. If the session is invalid, we will run a macro to log the user back in. We need to record a further macro, to perform the actual login, and tell Burp to run this macro and update the session handling cookie jar based on the results:
At this point, we have configured Burp to update requests with cookies from its cookie jar, and to log the user back in to the target application when their session is invalid. To complete the required configuration, we need to define a further rule to deal with the anti-CSRF token used in the function we want to test. The request we are testing looks like this:
POST /auth/4/NewUserStep2.ashx HTTP/1.1
To ensure that our requests to this function are properly handled, we need to ensure that a valid nonce is supplied with each request. The value of this nonce is supplied by the application in a hidden field within the form that generates the above request. So our rule needs to run a macro to fetch the page containing the form, and update the current request with the value of the nonce parameter. We add a further rule with an action of the type "run macro" and configure it as follows:
In the above configuration, we have specified that Burp should run a new macro, which fetches the form containing the anti-CSRF token, and then obtain the nonce parameter from the (final) macro response, and update this in the request. Alternatively, we could select the "update all parameters" option, and Burp would automatically attempt to match parameters in the request with those specified in the macro response.
In terms of the scope for this rule, this obviously needs to be defined more narrowly than the whole application domain. For example, we could define the rule to apply only to the exact URL in the above request. This is the best option if the application only employs anti-CSRF tokens in a few locations. However, in some applications, tokens are used for a large number of functions, and a token obtained within one function can be submitted within a different function. In this situation, we could define a rule that applies to the whole domain, but only to requests containing a specified parameter name. In this way, any time a request is made to the application that contains an anti-CSRF token, the rule will execute and Burp will fetch a new valid token to use in the request.
The full configuration, with its three session handling rules and three macros, looks like this within the main Burp UI:
You can test the configuration is working by logging out of the application, sending the authenticated, token-protected request to Burp Repeater, and verifying that it performs the required action. The request will probably take longer to return than normal, because behind the scenes Burp is making several other requests, to validate your session, log in again if necessary, and obtain a token to use in the request. Once you are happy that all of this is working correctly, you can send the request to Burp Intruder or Scanner, to perform your automated testing in the normal way.
Thursday, 24 March 2011
A key part of Burp's new session handling functionality is the ability to run macros, as defined in session handling rules. A macro is a predefined sequence of one or more requests. Typical use cases for macros include:
Fetching a page of the application (such as the user's home page) to check that the current session is still valid.
Performing a login to obtain a new valid session.
Obtaining a token or nonce to use as a parameter in another request.
When scanning or fuzzing a request in a multi-step process, performing the necessary preceding requests, to get the application into a state where the targeted request will be accepted.
Macros are recorded using your browser. When defining a macro, Burp displays a view of the Proxy history, from which you can select the requests to be used for the macro. You can select from previously made requests, or record the macro afresh and select the new items from the history.
When you have recorded the macro, the macro editor shows the details of the items in the macro, which you can review and configure as required:
As well as the basic sequence of requests, each macro includes some important configuration about how items in the sequence should be handled, and any interdependencies between items:
For each item in the macro, the following settings can be configured:
Whether cookies from the session handling cookie jar should be added to the request.
Whether cookies received in the response should be added to the session handling cookie jar.
For each parameter in the request, whether it should use a preset value, or a value derived from a previous response in the macro.
The ability to derive a parameter's value from a previous response in the macro is particularly useful in some multi-stage processes, and in situations where applications make aggressive use of anti-CSRF tokens. When a new macro is defined, Burp tries to automatically find any relationships of this kind, by identifying parameters whose values can be determined from the preceding response (form field values, redirection targets, query strings in links, etc.). You can easily review and edit the default macro configuration applied by Burp before the macro is used. Further, the configured macro can be tested in isolation, and the full request/response sequence reviewed, to check that it is functioning in the way you require.
Of course, the full power of using macros is only realised once they are incorporated into suitable session handling rules, to control the way that different requests are processed by Burp's tools, and work with the session handling mechanism and related functionality being used by the target application. As is perhaps already apparent, the configuration required in many real-world situations is complex, and mistakes are easily made. There is a need for a full in-tool debugger for troubleshooting the session handling configuration. In the meantime, an effective workaround is to chain a second instance of Burp as an upstream proxy from the instance being configured. The proxy history in the upstream instance will show you the full sequence of requests and responses that occur when your session handling rules are executed, helping you to find and fix any problems in your configuration.
Wednesday, 23 March 2011
Some problems commonly encountered when performing any kind of fuzzing or scanning of web applications are:
The application terminates the session being used for testing, either defensively or for other reasons, and the remainder of the testing exercise is ineffective.
Some functions use changing tokens that must be supplied with each request (for example, to prevent request forgery attacks).
Some functions require a series of other requests to be made before the request being tested, to get the application into a suitable state for it to accept the request being tested.
All of these problems can also arise when you are testing manually, and resolving them manually is often tedious, reducing your appetite for further testing.
The second broad area of new functionality in Burp v1.4 is a range of features to help in all of these situations, letting you continue your manual and automated testing while Burp takes care of the problems for you in the background. This functionality is quite complex, with a lot of configuration to explain. We'll start by looking at the core session handling features.
Firstly, Burp's cookie jar, which was previously part of the Spider tool, is now more sophisticated and is shared between all tools. Cookies set in responses are stored in the cookie jar, and can be automatically added to outgoing requests. All of this is configurable so, for example, you can update the cookie jar for cookies received by the Proxy and Spider, and have Burp automatically add cookies to requests sent by the Scanner and Repeater. The cookie jar configuration is shown in the new "sessions" tab within the main "options" tab:
As shown, by default the cookie jar is updated based on traffic from the Proxy and Spider tools. You can view the contents of the cookie jar and edit cookies manually if you wish:
For all tools other than the Proxy, HTTP responses are examined to identify new cookies. In the case of the Proxy, incoming requests from the browser are also inspected. This is useful where an application has previously set a persistent cookie which is present in your browser, and which is required for proper handling of your session. Having Burp update its cookie jar based on requests through the Proxy means that all the necessary cookies will be added to the cookie jar even if the application does not update the value of this cookie during your current visit.
Burp's cookie jar honours the domain scope of cookies, in a way that mimics Internet Explorer's interpretation of cookie handling specifications. Path scope is not honoured.
In addition to the basic cookie jar, Burp also lets you define a list of session handling rules, which give you very fine-grained control over how Burp deals with an application's session handling mechanism and related functionality. These rules are configured in the new "sessions" tab:
Each rule comprises a scope (what the rule applies to) and actions (what the rule does). For every outgoing request that Burp makes, it determines which of the defined rules are in-scope for the request, and performs all of those rules' actions in order (unless a condition-checking action determines that no further actions should be applied to the request).
The scope for each rule can be defined based on any or all of the following features of the request being processed:
The Burp tool that is making the request.
The URL of the request.
The names of parameters within the request.
Each rule can perform one or more actions. The following actions are currently implemented:
Add cookies from the session handling cookie jar.
Set a specific cookie or parameter value.
Check whether the current session is valid, and perform sub-actions conditionally on the result.
Run a macro.
Prompt the user for in-browser session recovery.
All of these actions are highly configurable, and can be combined in arbitrary ways to handle virtually any session handling mechanism. Being able to run arbitrary macros (defined request sequences), and update specified cookie and parameter values based on the result, allows you to automatically log back in to an application part way through an automated scan or Intruder attack. Being able to prompt for in-browser session recovery enables you to work with login mechanisms that involve keying a number from a physical token, or solving a CAPTCHA-style puzzle.
By creating multiple rules with different scopes and actions, you can define a hierarchy of behaviour that Burp will apply to different applications and functions. For example, on a particular test you could define the following rules:
For all requests, add cookies from Burp's cookie jar.
For requests to a specific domain, validate that the current session with that application is still active, and if not, run a macro to log back in to the application, and update the cookie jar with the resulting session token.
For requests to a specific URL containing the __csrftoken parameter, first run a macro to obtain a valid __csrftoken value, and use this when making the request.
We'll be examining some more details of how this functionality works shortly. In the meantime, it is worth noting a few points about how the new session handling features affect some of Burp's existing functionality:
There is a default session handling rule which updates all requests made by the Scanner and Spider with cookies from Burp's cookie jar. This changes the session handling of requests made by the Scanner. Previously, these requests were always made within the same session as the relevant base request - in other words, cookies appearing in the request that was sent to be scanned were used for all scan requests for that item. Now, different cookies may be used if Burp's cookie jar has been updated since the request was sent to be scanned. This change is normally desirable, as it ensures that all scan requests are made in-session, provided you maintain a valid session using your browser. It also means that items in the active scan queue that are loaded from a state file will be scanned within your current session, not the session that was active when the state file was saved. If this is not the behaviour you require, you should disable the default session handling rule before performing any scanning.
In previous versions of Burp, the Spider configuration included some basic settings for using cookies from the cookie jar. These settings have now been removed, and the Suite-wide session handling functionality should be used instead. The previously default behaviour for the Spider's handling of cookies is replicated by the new default session handling rule, as described above.
Similarly, Intruder previously included an option to run a specified request before each attack request, to obtain a new cookie to use in the attack request. This option has now been removed, and instead you should define a session handling rule to run a macro before each request (or, more likely, use some more elegant configuration to achieve the same objective).
In cases where session handling rules modify a request before it is made (for example, to update a cookie or other parameter), some of Burp's tools will show the final, updated request, for purposes of clarity. This applies to the Intruder, Repeater and Spider tools. Requests that are shown within reported Scanner issues continue to show the original request, to facilitate clear comparison with the base request, where necessary. To observe the final request for a scan issue, as modified by the session handler, you can send the request to Burp Repeater and issue it there.
When the Scanner or Intruder makes a request that manipulates a cookie or parameter that is affected by a session handling action, the action is not applied to that request, to avoid interfering with the test that is being performed. For example, if you are using Intruder to fuzz all the parameters in a request, and you have configured a session handing rule to update the "sessid" cookie in that request, then the "sessid" cookie will be updated when Intruder is fuzzing other parameters. When Intruder is fuzzing the "sessid" cookie itself, Burp will send the Intruder payload string as the "sessid" value, and will not update it as is done normally.
As I said, the new session handling is powerful and complex. In the next couple of posts, we'll look at how it works in more detail.
Tuesday, 22 March 2011
In the previous post, we described how the "compare site maps" feature can be used to automate much of the laborious work involved in testing access controls. In some situations, however, performing a wholesale comparison like this may not meet your needs. It may be that you prefer to work in a more piecemeal way, individually testing the controls over a small number of key requests. Further, where a sensitive action is performed using a multi-stage process, simply re-requesting an entire site map in a different user context many be ineffective. In this situation, to perform an action, the user must typically make several requests in the correct sequence, with the application building up some state about the user’s actions as they do so. Simply re-requesting each of the items in a site map may fail to replicate the process correctly, and so the attempted action may fail for reasons other than the use of access controls.
For example, consider an administrative function to add a new application user. This may involve several steps, including loading the form to add a user, submitting the form with details of the new user, reviewing these details, and confirming the action. In some cases, the application may enforce access controls over some of these steps but not others. For example, the application may protect access to the initial form, but fail to protect the page that handles the form submission, or the confirmation page. The overall process may involve numerous requests, including redirections, with parameters submitted at earlier stages being retransmitted later via the client side. Each step of this process needs to be tested individually, to confirm whether access controls are being correctly applied.
The new version of Burp contains several features to facilitate this kind of testing. Firstly, when you are testing multi-stage functions in different user contexts, it is often helpful to review side-by-side the sequences of requests that are made by different users, in order to identify subtle differences that may merit further investigation. Burp now lets you:
Filter the proxy history based on the local proxy listener.
Open additional proxy history windows, each with its own filter.
To use these features to help test access controls, you need to use a separate browser for each user context you are testing, and create a separate proxy listener in Burp for use by each browser (you will need to update your proxy configuration in each browser to point to the relevant listener). For each browser, you can then open a separate proxy history window in Burp, and set the filter to show only requests from the relevant proxy listener. As you use the application in each browser, each history window will show only the items for the associated user context.
To open a new proxy history window, use the context menu on the main proxy history tab:
So far, this just gives you a way of separating the series of requests made by different user contexts. To actually evaluate access controls over a multi-stage process, you need to test each stage individually, reissuing the request in a different user context, and determining how this is handled by the application. This can often be a laborious process, and involves walking through the multi-stage process repeatedly using your browser, intercepting requests using the proxy, and updating the cookie in one or more requests so that they are issued in a different user context.
A further feature in the new version of Burp takes away some of this pain. It lets you select a request anywhere within Burp, and reissue the identical request from within your browser, within the browser's current session with the application. Using this feature, you can test access controls over a multi-stage process in the following way:
In your Burp proxy history, find the series of requests that are made when a higher privileged user performs the multi-stage process.
For each request, select the "request in browser in current session" option from the context menu. This gives you a unique URL to paste into your browser.
Paste the URL into the address bar of a browser that is logged in to the application as a lower privileged user who should not be able to perform the action. (This browser must be configured to use Burp as its proxy.)
If the application lets you, follow through the remainder of the multi-stage process in the normal way, using your browser.
Review the results to determine whether the lower privileged user was successful in performing the action.
This methodology, of manually pasting a series of URLs from Burp into your browser, is normally a lot easier than repeating a multi-stage process over and over, and modifying cookies manually using the proxy. In general, the new feature provides a highly efficient means of grabbing a request from the site map, or proxy history, or anywhere else, and repeating the request within your current browser session, to see how it is handled.
For people who are interested, the new feature is implemented as follows. When your browser requests the unique URL provided by Burp, Burp returns a redirection to the URL in the original request that you selected. When your browser follows the redirection, Burp replaces the outgoing request with the full request that you originally selected (headers and body), with the exception of the Cookie header, which is unmodified. When the browser receives the application's response to this request, it processes it in the context of the original URL, so all relative links, DOM access, etc. work correctly. Neat, huh?
Monday, 21 March 2011
Somewhat later than planned, as is customary, Burp v1.4 is nearly ready, and it's time to share with you the highlights of what is coming. This release focuses on a small number of frequently requested features which, though you may not use them every day, can in some situations really make your life easier. Over the next few days, I'll be blogging about different features, to whet your appetite. Then I'll release a beta version for Pro users to play with. Everyone with a current license will receive an automatic upgrade to the new version.
The first broad area of new functionality in Burp v1.4 is various features to help test access controls. Fully automated tools generally do a terrible job of finding access control vulnerabilities, because they do not understand the meaning or context of the functionality that is being tested. For example, an application might contain two search functions - one that returns extracts from recent news articles, and another that returns sensitive details about registered users. These functions might be syntactically identical - what matters when evaluating them is the purpose of each function and the nature of the information involved. These factors are way beyond the wit of today's automated tools.
Burp does not try to identify any access control bugs by itself. Instead, it provides ways of automating much of the laborious work involved in access control testing, and presents all of the collected information in a clear form, allowing you to apply your human understanding to the question of whether any actual vulnerabilities exist.
One exciting new feature to help with access control testing is the facility to compare two site maps and highlight differences. This feature can be used in various ways to help find different types of access control vulnerabilities, and identify which areas of a large application warrant close manual inspection. Some typical use-cases for this functionality are as follows:
You can map the application using accounts with different privilege levels, and compare the results to identify functionality that is visible to one user but not the other.
You can map the application using a high-privileged account, and then re-request the entire site map using a low-privileged account, to identify whether access to privileged functions is properly controlled.
You can map the application using two different accounts of the same type, to identify cases where user-specific identifiers are used to access sensitive resources, and determine whether per-user data is properly segregated.
You can access the new feature using the context menu on the main site map:
This opens a wizard that lets you configure the details of the site maps you want to compare, and how the comparison should be done. When selecting the site maps you want to compare, the following options are available:
The current site map that appears in Burp's target tab.
A site map loaded from a Burp state file that you saved earlier.
Either of the above, re-requested in a different session context.
You can choose to include all of the site map's contents, or you can restrict only to selected or in-scope items. If you choose to re-request a site map in a different session context, it is particularly important not to include requests that might disrupt that context - for example, login, logout, user impersonation functions, etc.
To perform the comparison, Burp works through each request in the first site map, and matches this with a request in the second site map, and vice versa. The responses to matched requests are then compared to identify any differences. Any unmatched items in either site map are flagged as deleted or added, respectively. The exact process by which this is done is highly configurable, allowing you to tailor the comparison to features of the target application.
The options for configuring how Burp matches requests in the two site maps are shown below:
The default options shown will work well in most situations, and match requests based on URL file path, HTTP method and the names of parameters in the query string and message body. For some applications, you will need to modify these options to ensure that requests are correctly matched. For example, if an application uses the same base URL for various different actions, and specifies the action using the values of query string parameters, you will need to match requests on the values of these parameters as well as their names.
The options for configuring how Burp compares the responses to matched requests are shown below:
Again, the default options will work in most situations. These options ignore various common HTTP headers and form fields that have ephemeral values, and also ignore whitespace-only variations in responses. The default options are designed to reduce the noise generated by inconsequential variations in responses, allowing you to focus attention on differences that are more likely to matter.
The results of a simple site map comparison are shown below. This shows an application that has been mapped out with administrative privileges, and the resulting site map re-requested with user-level privileges. The results contain a colourised analysis of the differences between the site maps, and show items that have been added, deleted or modified between the two maps. (In this case, since the whole of the first site map was re-requested, there are no added or deleted items in the maps themselves.) For modified items, the table includes a “diff count” column, which is the number of edits required to modify the item in the first map into the item in the second map. When you select an item, the corresponding item in the other site map is also selected, and each response is highlighted to show the locations of the differences:
Interpreting the results of the site map comparison requires human intelligence, and an understanding of the meaning and context of specific application functions. For example, the screenshot above shows the responses that are returned to each user when they view their home page. The two responses show a different description of the logged-in user, and the administrative user has an additional menu item. These differences are to be expected, and they are neutral as to the effectiveness of the application’s access controls, since they only concern the user interface.
The screenshot below shows the response returned when each user requests the top-level admin page. Here, the administrative user sees a menu of available options, while the ordinary user sees a “not authorised” message. These differences indicate that access controls are being correctly applied:
The screenshot below shows the response returned when each user requests the “list users” admin function. Here, the responses are identical, indicating that the application is vulnerable, since the ordinary user should not have access to this function and does not have any link to it in their user interface:
As this example shows, simply exploring the site map tree and looking at the number of differences between items is not sufficient to evaluate the effectiveness of an application’s access controls. Two identical responses may indicate a vulnerability (for example, in an administrative function that discloses sensitive information), or may be harmless (for example, in an unprotected search function). Conversely, two different responses may still mean that a vulnerability exists (for example, in an administrative function that returns different content each time it is accessed), or may be harmless (for example, in a page showing profile information about the currently logged-in user). All of these scenarios may coexist even in the same application. This is why fully automated tools are so ineffective at identifying access control vulnerabilities.
So Burp does not relieve you of the task of closely examining the application's functionality, and evaluating whether access controls are being properly applied in each case. What the site map comparison feature does is to automate as much of the process as possible, giving you all the information you need in a clear form, and letting you apply your knowledge of the application’s functionality to identify any actual vulnerabilities.