Thursday, April 16, 2015

Introducing Burp Collaborator

Today's release of Burp Suite introduces Burp Collaborator. This new feature has the potential to revolutionize web security testing. Over time, Burp Collaborator will enable Burp to detect issues like blind XSS, server-side request forgery, asynchronous code injection, and various as-yet-unclassified vulnerabilities.

In the coming months, we will be adding many exciting new capabilities to Burp, based on the Collaborator technology. See the "Release roadmap" section towards the end of this post for more details.

This blog post looks at:
  • Some important limitations of the conventional web testing model.
  • How Burp Collaborator will address these limitations.
  • Lots of real-world vulnerabilities that don't fit into the usual classifications.

The conventional web testing model

At its core, conventional web security testing is fairly simple: we send payloads to the target application, and analyze responses to identify security vulnerabilities.

This model is well established and understood, and enables us to identify many kinds of security issues. Much of the work can also be automated with a degree of reliability.

However, the conventional testing model also misses lots of bugs, for example:
  • We miss "super-blind" injection vulnerabilities. People often refer to "blind SQL injection", meaning only that an application doesn't return helpful error messages when a payload breaks the interpreted query. But in some situations, a successful injection has no detectable effect on application responses: that is, no difference in response contents or timing. For example, SQL injection into an asynchronous logging function is typically super-blind in this sense.
  • We miss many vulnerabilities involving stored data. Bugs like stored XSS are easy to find in principle (by monitoring later responses for earlier payloads). But other stored bugs are much harder to find. For example, in stored (or second-order) SQL injection, data placed safely into a database is later read back and concatenated into a SQL query. To find bugs like this in the conventional model, we might need to brute force every combination of requests within the application, to send a payload in request 1 and look for effects via request 2.
  • We miss vulnerabilities where a successful attack can only be detected in areas of the application that are not directly visible to the tester. For example, stored XSS vulnerabilities affecting admin-only pages.
  • We miss many vulnerabilities involving interaction with external systems. Bugs in this category are often given labels like "server-side request forgery" and "remote file include".

Enter Burp Collaborator

Burp Collaborator augments the conventional testing model with a new component, distinct from Burp and the target application. Burp Collaborator can:
  • Capture external interactions initiated by the target that are triggered by Burp's attack payloads.
  • Deliver attacks back against the target in responses to those interactions.
  • Enable the reliable detection of many new vulnerabilities.

The basic design of Burp Collaborator includes the following features:
  • The Burp Collaborator server runs on the public web (by default).
  • It uses its own dedicated domain name, and the server is registered as the authoritative DNS server for this domain.
  • It provides a DNS service that answers any lookup on its registered domain (or subdomains) with its own IP address.
  • It provides an HTTP/HTTPS service, and uses a valid, CA-signed, wildcard SSL certificate for its domain name.
  • In future, custom implementations of various other network services will be provided, such as SMTP and FTP.

Detecting external service interaction

External service interaction occurs when a payload submitted to the target application causes it to interact with an arbitrary external domain using some network protocol:

This behavior is sometimes referred to as "server-side request forgery". We prefer the term "external service interaction" because this captures more general behavior: interactions can be triggered using protocols other than HTTP, such as SMB or FTP.

External service interaction can represent a serious vulnerability because it can allow the application server to be used as an attack proxy to target other systems. This may include public third-party systems, internal systems within the same organization, or services available on the local loopback adapter of the application server itself. Depending on the network architecture, this may expose highly vulnerable internal services that are not otherwise accessible to external attackers.

The Burp payload includes a random subdomain of the main Collaborator domain. When an HTTP-based external service interaction occurs, the Collaborator server will typically receive a DNS lookup for this subdomain, followed by an HTTP request. However, it is noteworthy that the DNS lookup alone is sufficient to report an issue. If a payload starting "http://..." causes a DNS-only interaction, then this strongly indicates that outbound HTTP from the server is being blocked at the network layer. In this situation, a follow-up attack could target other systems internal to the organization or running on the server's loopback adapter. For this reason, Burp separately reports any DNS and HTTP interactions that are triggered.

In its issue advisory, Burp reports the in-band request that was made to the application, and the full details of the interactions that occurred with the Collaborator server:

Detecting out-of-band resource load

Out-of-band resource load occurs when a payload submitted to the target application causes it to fetch content from an arbitrary external domain using some network protocol, and incorporate that content into its own response to the request that contained the payload:

This behavior is sometimes referred to as "remote file include". However, this label has connotations of vulnerabilities like PHP remote script inclusion. We prefer the term "out-of-band resource load" because this captures the more general behavior where content obtained out-of-band is somehow incorporated into the application's own responses. (Server-side execution of content obtained out-of-band is discussed later in this post.)

Out-of-band resource load is a potentially high-risk issue. An attacker can leverage the application as a two-way attack proxy, sending arbitrary payloads to, and retrieving content from, other systems that the application server can interact with. Again, this may include third-party systems or sensitive internal systems that are otherwise inaccessible.

Additionally, the application's processing of out-of-band content exposes some important and non-conventional attack surface. The possible ways of targeting this are described in more detail later in this post.

Burp reports the full details of the interactions that occurred with the Collaborator server, showing how the content returned from the Collaborator is propagated back in the application's in-band response to the user:

Detecting out-of-band XSS

In cases where out-of-band resource load is identified, the most obvious related issue to check for is out-of-band XSS:

Out-of-band XSS doesn't fall under the usual XSS classifications:
  • Not reflected: the payload isn't reflected from the current in-band request to the target application.
  • Not stored: the payload isn't (necessarily) stored by the target application.
  • Not DOM-based: existing JavaScript isn't (necessarily) involved.
Out-of-band versions of some other client-side issues arise in a similar way, including HTTP response header injection, and open redirection.

Detecting "super-blind" injection bugs

In cases where a code injection attack cannot trigger any detectable effect in the application's own responses (that is, any difference in response contents or timing), we can often detect the vulnerability by using payloads that trigger an external service interaction when the injection is successful:

Viable payloads exist for many categories of injection vulnerabilities, including SQL injection, XXE injection, OS command injection, etc. With these payloads, Burp does not need to observe any effect in the application's own responses, and successful injection can be detected solely through the fact that the external interaction occurred. Note also that in this situation, only a DNS interaction is needed to confirm the injection. Since outbound DNS is almost always permitted because servers need to use it, network-layer filters are unlikely to hinder detection.

Detecting out-of-band injection bugs

When any type of external service interaction is identified, we can use the Collaborator server's responses to deliver conventional input-based payloads back to the application. Depending on how the application processes the Collaborator's responses, potentially any input-based vulnerability could exist, including SQL injection and server-side code execution:

Vulnerabilities like this might be surprisingly common because they are not thoroughly tested for at present.

Detecting stored out-of-band bugs

In addition to the out-of-band injection bugs described in the previous section, there is the potential for stored out-of-band attacks in any situation where the target application processes data that it receives in responses from the Collaborator server.

Detecting stored out-of-band resource load is straightforward:

Based on the above behavior, we can also test for stored out-of-band XSS:

Whenever we discover stored out-of-band resource load, we are able to link an entry point and a retrieval point within the application. We know that the application is storing and processing data between the two locations, so we can check for stored out-of-band versions of any type of input-based bug:

Detecting blind stored vulnerabilities

We already described how we can detect "super-blind" injection bugs using payloads that trigger an external service interaction when the injection is successful. We can also detect cases where these (or other) payloads are stored and processed later in an unsafe way. Deferred interactions with the Collaborator server enable us to report blind stored versions of many input-based bugs. 

For example, blind stored XSS arises when the result of a stored XSS attack cannot be detected from the tester's perspective, because the tester does not have access to the area of the application where the stored attack executes. We can detect these issues by submitting stored XSS payloads that will trigger an interaction with the Collaborator server when the attack later executes:

In this example, when an application administrator visits the relevant protected page, the stored attack executes, and Burp can confirm this via the Collaborator server. Further, the HTTP Referer header in the administrator's incoming request to the Collaborator lets us identify the location of the XSS retrieval page, which we cannot directly see ourselves.

Reporting deferred interactions

In general, any Collaborator-related payload that Burp sends to the target application might cause deferred interactions with the Collaborator server. This can happen in two main ways:
  • Conventional storage and later processing of input, e.g. stored SQL injection.
  • Immediate asynchronous processing, e.g. by a mail spooler.
When Burp polls the Collaborator server to retrieve details of any interactions that were triggered by a given test, it will also receive details of any deferred interactions that have resulted from its earlier tests. Burp can then report the relevant issues to the user retrospectively.

Because every Collaborator payload that Burp sends to the target includes a unique, one-time random identifier, when a deferred interaction occurs, Burp can use the identifier to pinpoint exactly where the payload originated, including the original request, the insertion point and the full payload. 

Imagine this: A day after you finish your test, you fire up Burp Suite and open the same project. Burp polls the Collaborator and reports some deferred-interaction issues, with all the usual details. You call up the client and tell them about all their blind stored XSS ...

Other network services

We have focused on vulnerabilities involving DNS and HTTP interactions between the target application and Burp Collaborator. In fact, we can use payloads to trigger Collaborator interactions using various other protocols. For example, we can potentially:
  • Inject into mail headers and detect this via SMTP interactions with the Collaborator.
  • Trigger a connection to an SMB share and capture the NTLM handshake.
  • Trigger a connection to an SSH service and capture credentials.

How prevalent are these issues?

Some of the application behavior we have described is relatively rare, for example:
  • Getting a URL from a request parameter.
  • Fetching the response from the URL.
  • Processing the response contents in some way.
Other behavior is relatively common, for example:
  • Storage of conventional user-supplied data.
  • Later reuse in a potentially unsafe manner.
  • Super-blind processing of results.
It seems fair to say that the cases where the relevant behavior exists are relatively untested, even by skilled attackers:
  • Full Collaborator-style functionality is expensive to create, requiring a custom domain name, wildcard SSL certificate, and custom implementations of various network protocols.
  • Even with a suitable Collaborator-like server deployed, manually correlating input entry points and results of stored and deferred interactions is time consuming and error-prone.
  • No automated scanner currently does everything that we've described.
Hence, we may reasonably conclude that where the relevant behavior exists, there is lots of low hanging fruit to be found. Anecdotal evidence from skilled bug bounty hunters supports this conclusion. (Sorry guys, Burp will soon open up these bugs to the masses!)

Manual testing tools

In the hands of a well-designed automated scanner, the Collaborator concept is incredibly powerful. There is also potential for some great manual tools that leverage the Collaborator's capabilities. These tools would let users manually verify and exploit issues that Burp Scanner has reported (for example, performing custom data exfiltration via super-blind SQL injection), and also do their own targeted testing of relevant attack surface.

Two key manual tools are planned:
  • Burp Collaborator client
  • Burp Intruder integration

Burp Collaborator client

This will include the following components
  • Monitoring function - This will generate a unique Collaborator identifier for you to use in your own test payloads. It will poll the Collaborator and give full details of any interactions that result from using this identifier.
  • Response-control function - This will let you configure the response that the Collaborator should return to targets that interact with it.
Used in conjunction with Burp Repeater, the Collaborator client will give you real-time three-way interaction between Burp, the target application, and the Collaborator server, all under full manual control.

Burp Intruder integration

This will provide a new Intruder attack option to use out-of-band payload delivery when an external service interaction has been identified. With the option enabled, Burp will send a unique Collaborator URL as the in-band payload to the target application. When the target performs the external interaction, the Collaborator server will return the actual configured payload to the target.

Security of Collaborator data

At this point, you may be forgiven for asking "Does this mean that the Collaborator server holds details of all the vulnerabilities that are found using it? What's to stop anyone else fetching the details of my issues?".

In fact, you're not just forgiven for asking this: we'd be disappointed if you didn't. We've spent a lot of time ourselves thinking about these questions.

What data does the Collaborator server store?

In most cases, when a vulnerability is found, the Collaborator server will not receive enough information to identify the vulnerability. It does not see the HTTP request that was sent from Burp to the target application. In a typical case, it will record that an interaction was received from somewhere, including a random identifier that was generated by Burp. Occasionally, the Collaborator server will receive some application-specific data: for example, the contents of an email generated through a user registration form.

How is retrieval of Collaborator data controlled?

The Collaborator functionality is designed so that only the instance of Burp that generated a given payload is able to retrieve the details of any interactions that result from that payload. This requirement is implemented as follows:
  • Each instance of Burp generates a securely random secret.
  • Each Collaborator-related payload that Burp sends to the target application includes a random identifier that is derived from a one-way hash (cryptographic checksum) of the secret.
  • Any resulting interactions with the Collaborator will include this identifier in the transmitted data (for example, in the subdomain of a DNS lookup, or the Host header of an HTTP request).
  • The secret is only ever sent by Burp to the Collaborator server, to poll for details of the resulting interactions. This is done using HTTPS, unless overridden in Burp's options.
  • When the Collaborator server receives a polling request, it performs the one-way hash of the submitted secret, and retrieves the details of any recorded interactions containing identifiers that are derived from that hash.
  • Hence, only the instance of Burp that generated the secret is able to retrieve the details of any interactions that are triggered by its payloads.
Further to this mechanism, the following precautions are also implemented in the Collaborator server to protect against unauthorized access to its data:
  • Details of interactions are stored in ephemeral process memory only.
  • No data of any kind is in recorded in any persistent form: for example, a database or log file.
  • Details of interactions are typically retrieved by Burp shortly after they occur, and are then discarded by the server.
  • Details of old interactions that have not been retrieved by Burp are discarded after a fixed interval.
  • There is no administrative function for viewing interaction details, only the retrieval mechanism already described.
  • The Collaborator server does not by design receive any data that could be used to identify any individual Burp user (such as an account name or license key).

Private Collaborator servers

Users who remain paranoid about any data passing through the shared Burp Collaborator server have two alternative options. If you wish, you can completely disable use of the Collaborator feature within Burp. Even better, you can choose to run your own private Collaborator server:

Anyone with a Burp Suite Professional license can run their own instance of the Collaborator server. To do this fully effectively, you will need a host server, a dedicated domain name, and a valid CA-signed wildcard SSL certificate. Private Collaborator servers without a suitable domain name or SSL certificate will be able to support some, but not all, of the Collaborator-related capabilities within Burp.

You can protect your private Collaborator instance at the network layer: you can configure different network interfaces to receive interactions and answer polling requests, and you can apply whatever IP restrictions you want, given the locations of your targets and testers. The option to use a private Collaborator server is likely to appeal to penetration testing firms and in-house security teams. It can also enable individual testers to deploy a Collaborator instance when testing on private closed networks with no Internet access. 

See the documentation on deploying a private Collaborator server for more details.

Release roadmap

Today's Burp Suite release includes the following elements:
  • Core Burp Collaborator platform and server.
  • Support for DNS, HTTP and HTTPS services.
  • Reporting of external service interaction and out-of-band resource load.
  • Support for private Collaborator servers.
In the coming months, we will deliver some of the other capabilities described in this post, including:
  • Out-of-band versions of all input-based scan checks.
  • Detection of various "super-blind" vulnerabilities.
  • Checks for stored versions of all relevant vulnerabilities.
  • Handling of deferred interactions and retrospective reporting of resulting issues.
  • Support for other network service protocols, and associated test payloads.
  • Manual testing tools.

Have fun!


Lucy said...

Can I have any provider name to get wildcard cert for my site? I don't want to pay much but want to cover 6-7 sub domains with main site.

Dafydd Stuttard said...

@Lucy Any SSL certificate authority will be able to sell you a wildcard SSL certificate.

HexBit said...

Really cool and innovative new feature! Love it!

Anonymous said...

Great work. You mention that interactions are stored in ephemeral process memory, but for how long exactly. If I fire up burp 3 days later and there is finally an interaction, has the server discarded the out-of-band data ?

Dafydd Stuttard said...

@Anonymous. At the moment, Burp doesn't support retrospective reporting of deferred interactions - this feature is planned for later this year. We think that storing unpolled interactions for up to two weeks would be reasonable balance between supporting users who take a break in testing a given target, and avoiding hoarding old data too long.

Sean de Regge said...

Great work as always Dafydd! Small comment: I think you meant "Super Blind XXE Injection" for example 4 at

Dafydd Stuttard said...

@Sean. Thanks!

Actually, the issue there is SQL injection. The payload uses a SQL query to execute a function that parses a supplied string as XML. Then, within that string, we define an external entity that triggers the Collaborator interaction. So the bug we've found is SQL injection - we're just leaning on an injected XML function to induce the interaction.

It's a nice payload, and works on recent versions of Oracle in a non-privileged user context, requiring only outbound DNS to report the issue.

Adrian Pastor said...

Indeed, traditional web app pentesting methods are limited and this post does an excellent job at explaining such limitations. The new Collaborator functionality will certainly be welcomed by the security-testing community. Keep up the good work, Dafydd & company!

Sean said...

This is a great feature and all, but why would you enable this by default?

Without knowing it, anyone who just upgraded to this version is now sending all of this data to a third party server (you). Does this not seem like a bad idea to anyone else?

You inform us about sending anonymous statistics about Burp usage when the program starts and give the option to disable- why don't you take the same approach to this much more sensitive data?

Dafydd Stuttard said...

@Sean. Regarding your comment "anyone who just upgraded to this version is now sending all of this data to a third party server" - this is extremely misleading.

Burp does not send details about discovered vulnerabilities to the Collaborator server in any circumstances. Rather, it sends to the target application payloads like:

This URL contains a meaningless random string as its subdomain. If the application has the behavior we are looking for, it will make an HTTP request to the specified domain. All the Collaborator server knows is that it has received an HTTP request containing this string. The originating instance of Burp (and no one else) can ask the Collaborator server whether it received the request, and if so Burp (not the Collaborator) identifies the issue.

The are significant differences between Burp's usage of the Collaborator server and submission of performance feedback:

- Performance feedback is sent from Burp to us. It contains certain (minimal, anonymous) information about problems with that instance of Burp. It is intended to be retrieved and analyzed by us. Sending feedback does not directly benefit the individual user. It lets us generally improve Burp for everyone.

- Burp does not submit any vulnerability information to the Collaborator. The information received by the Collaborator server cannot, by design, be accessed by anyone other than the Burp user themselves. Using the Collaborator feature directly benefits the individual user by giving Burp significantly enhanced scanning capabilities.

So yes, we have enabled usage of the Collaborator by default, and made significant efforts to protect the data it receives. We haven't sneaked this change out - we've made more noise about this release than any other for a long time. And we've provided an easy way for people to deploy their own private Collaborator server if they prefer.

Anonymous said...

Hi i'm getting an hostname error for the randomly generated subdomain. I'm the public server. Is this an issue? Thanks

Sean said...

@Dafydd Stuttard

Thank you for addressing my comment, and I understand you took great pains to anonymize and secure the data as much as possible, but I still disagree with the 'on by default' roll-out. However it is an easy option to disable.

If the Collaborator server receives requests to those specific domains, does the server know/keep track of where those requests come from (ie domain names, referrer headers)? This would reveal, in essence, web apps that were being tested, and therefor the companies. My customers would not like that very much. This is my concern.

However, let me say you have provided the ability to host our own server free of charge, which is absolutely *awesome*. Most companies would charge extra for this option, and as usual you have kept the price of Burp lower than all of your competitors by a wide margin and continue to provide extra features at a rapid rate. Burp is IMHO the best tool out there for testing web apps, and the most reasonably priced. Love your tool, I just don't agree with the configuration option.

Please keep being awesome :)

Dafydd Stuttard said...

@Sean It would be exceptionally unlikely for server-side HTTP/S interactions with the Collaborator to contain Host or Referer headers that identify the application being tested:

- If present, the Host header should contain a subdomain of the Collaborator domain, since that is the destination for the request that has been made.

- The Referer header is added by browsers when links are followed. Since server-side interactions with the Collaborator are not initiated by a browser, by rather directly by server-side application code, they will not normally contain a Referer header.

In future, we plan to implement support for blind stored XSS detection, which will allow Burp to identify stored XSS against parts of the application to which the tester does not have direct access. The stored XSS payloads will cause Collaborator interactions that are caused by a browser, not the server, and so they sometimes will include a Referer header that identifies the application being tested. As with all interaction data, this is protected and only retrievable by the instance of Burp that is performing the testing. But it is a factor to consider when deciding whether the public Collaborator or a private instance is the best option for you.

Yes, we definitely think making it free to deploy private Collaborator servers is the right decision. We want this to be as easy as possible for people who are paranoid about using the public one.

Anonymous said...

I know this is an old post but I am curious about the sequence of messages displayed here... is collaborator responding to the request with the javascript alert? If so, how does collaborator know to respond with the javascript alert box (instead of the normal garbage string response)? Does burp scanner first send a control message to collaborator to enable XSS responses (this pic appears to show control messages from scanner to collaborator)? If so, can you tell me what the control message api is that collaborator listens for? I am curious about how an attacker might misuse my private collaborator server to serve up javascript or any other user controlled content.

Also, can you clarify what the "garbage string" mentioned above actually is? I assume it is somehow related to the burp secure key used to fetch results from collaborator?

Dafydd Stuttard said...

I take it you are referring to the "Out-of-band XSS" section, where XSS payloads are returned from the Collaborator. This isn't currently implemented. When we do so, it is most likely that the "reverse" payloads will be triggered by an opcode-like identifier within the interaction ID, rather than a preceding control message from Burp to configure the payload to respond with.

The "garbage" subdomains aren't just garbage. They incorporate an identifier that is derived from Burp's secret key for that session, and a per-payload identifier that enables Burp to later determine where the payload was sent (what request, parameter, etc.). They aren't meaningful to anything other than the instance of Burp that generated them.

Anonymous said...

Thanks for the reply. For the "garbage" string, I am referring to the response collaborator sends the app under test after the subdomain is called. So, burp scanner injects the key hash string into the app under test as a subdomain, the app ends up making a GET request on the subdomain, and in response burp collaborator responds with a "garbage" string that I assume is somehow related to the key hash or maybe message request checksum? Just wanting a better picture of how this works and what all info collaborator serves up before we deploy our own server. Thanks again!

Dafydd Stuttard said...

Ok, the "garbage" string in the Collaborator response is just a deterministic mutation of the interaction ID that it received. Burp knows the mutation algorithm so is able to spot Collaborator response data coming back in in-band responses. This triggers the issue we call "Out-of-band resource load".

Anonymous said...


Dan said...

Hi Dafydd,

I'm not sure if I am just missing it or if it never got released, but what happened to the manual collaborator usage with repeater and the intruder integration? I remember thinking at the time that it would be useful and I'm now in a situation where I could use it but I can't work out a way to do so!

Kind Regards,

Dafydd Stuttard said...

@Dan. The manual Collaborator features aren't yet done, sorry. We've been working on all of the native automated capabilities that we want to layer over the core Collaborator functionality. Once that is done, we'll move on to manual testing tools. We definitely plan to deliver these, and believe they will be phenomenally useful to manual testers. Bear with us!