DonsProxy HowTo - Using the HttpXmlSubscriber to analyze site characteristics

Follow the Basic example, but instead of running httphead.sh, run the script httpxml.xml:

./httpxml.sh
2008-03-09 08:09:24,365 INFO  [HttpXmlSubscriber] - Writing XML output to HttpXmlSubscriber.xml
2008-03-09 08:09:24,365 INFO  [HttpXmlSubscriber] - HttpXmlSubscriber started.

All output will be written to HttpXmlSubscriber.xml in the current directory. Okay, here's the "tricky" part - I haven't modified HttpXmlSubscriber yet to catch the exit, so for now it does its cleanup when the proxy shuts down. So to use it properly:

Run proxy.sh
Run httpxml.sh
Collect your output as usual, and XML will be written to HttpXmlSubscriber.xml
Kill the proxy

The last step will cause HttpXmlSubscriber to stop as well, and in the process it will finish writing the XML output. The class com.moneybender.proxy.analyzers.CachingSummary is an example of how the XML can be processed using a SAX parser such as Xerces. The subscriber has already done the "hard" work of pairing together the requests and their responses. The analyzer just loops through the XML and generates a human-readable summary in html form:

./sessionsummary.sh
2008-03-09 08:23:55,505 INFO  [CachingSummary] - Processing file HttpXmlSubscriber.xml
2008-03-09 08:23:55,521 INFO  [CachingSummary] - Analyzed HttpXmlSubscriber.xml and left results in HttpXmlSubscriber.html

Since the output is in XML, there is a veritable smorgasbord of parsing tools available to process the results in whatever way you see fit. The old DonsProxy included some XSLT transformations that would convert the output to Perl and bash scripts that could be used to repeat the session for testing and monitoring of a web site. Yes, you can use XSLT, but it can get pretty squirrelly to generate Perl from XML using XSLT. To me, SAX parsing is more straightforward, but that's just a matter of preference.

For illustration, let's look at a sample Ruby on Rails application, eXPlain Project Management Tool. A quick run using under Mongrel and under Webrick will let us see any minor differences. A longer run might let us analyze performance. If you're not familiar with Ruby or Ruby on Rails, don't worry about it. The same basic process applies regardless of what you use to implement your application.

The first step is to run eXPlain under Mongrel, and collect the output:

ruby script/server

eXPlain will now be running on port 3000. Next, modify proxy.sh to listen on 3001, and forward its traffic to 3000:

...
PROXY_PROPS="-Dlisten.port=3001 -Dtarget.host=localhost -Dtarget.port=3000"
...

Run HttpXmlSubscriber to collect the data:

./httpxml.sh
2008-03-09 22:28:01,271 INFO  [HttpXmlSubscriber] - Writing XML output to HttpXmlSubscriber.xml
2008-03-09 22:28:01,333 INFO  [HttpXmlSubscriber] - HttpXmlSubscriber started.

Use a browser to hit http://localhost:3001, and performance a few operations. Once complete, go to the window running proxy.sh, and kill it. HttpXmlSubscriber will also stop, and flush its data to HttpXmlSubscriber.xml, where we can analyze it. You can expect that it would look something like this: MongrelExplain.xml. Now process the data. Run CachingSummary by invoking the script ./sessionsummary.sh, and it will create an output file that looks something like this: MongrelExplain.html

We might expect to see different results if we run under Webrick, and we do, but this isn't enough data to draw any conclusions. You should run a longer test before trying to decide what it means. If you'd like to save the results from this run, copy HttpXmlSubscriber.xml and HttpXmlSubscriber.html before proceeding.

Now, stop the process running eXPlain, and restart it under Webrick:

ruby script/server webrick

Collect the data in the same manner as before. You can see my results in WebrickExplain.xml and in WebrickExplain.html

Notice that Mongrel is delivering javascript as "text/javascript" and Webrick delivers it as "application/octet-stream". Interesting. You can see that there are some performance differences, but again, there's not enough data here to draw conclusions.