Monday, November 10, 2008

[MoBP] The all new Burp Spider

In its current incarnation, the Spider is the weakest of the core Burp tools, with more than its fair share of old buggy code, and several obstacles to usability. I don't use it much myself, and I doubt if too many of you do either.

In the new release, Spider has been completely rewritten from scratch, with much improved content parsing and several new features. Spidering is now driven entirely via the target site map and other tools.

When you first map out a new application's content and functionality, it is generally best to work manually with your browser, giving you full control over the requests that are issued, and ensuring that you comply with any input validation, navigational structure and other constraints imposed on normal usage of the application. As you do this, Burp will passively compile its site map of all the items you have requested, as well as those which it has inferred from the application's responses.

When you have explored all of the content you can find with your browser, you will typically see a site map containing a number of unrequested items - these are shown in grey in the tree and table. At this point, you can still proceed manually, copying the relevant URLs into your browser and exploring further. Or you can let the Spider do its work to map out the rest of the application's content. The easiest way to do this is to select one or more nodes within the tree, and choose "spider from here" from the context menu:

When you tell Burp to spider a branch of the site map, it will perform the following actions:

  • Request any unrequested URLs identified within the branch.

  • Submit any forms whose action URLs lie within the branch.

  • Re-request any items which previously returned 304 status codes, to retrieve a fresh (uncached) copy of the application's responses.

  • Parse all content retrieved to identify new URLs and forms.

  • Recursively repeat these steps as new content is identified.


Jesper said...

One other thing, what about the pro-version? Are there any changes that only applies to those who runs that version? Are the changes included in the licence?

PortSwigger said...

Everything I've discussed so far is in the free version. When I come on to discussing some of the new pro features, I'll make it clear.

Alex said...

So if you have the professional version of Burp there are no extra licens costs related to the upgrade?

About the spider, let's say I want to spider a particular web application for swf files and later download them. Is there a way to sort the MIME type of swf and export the URLs or something like that?

PortSwigger said...


You can sort by any column in the results table, multi-select interesting items, and copy the list of URLs using the context menu. That will hopefully do what you require.

Anonymous said...

I'm missing the "linked from" window. Did you leave it out or where do I've got to look for it.