Preparing to optimize Mundlejs

Update- Blog moved to: http://saleemabdulhamid.com/blog/2012/6/preparing-to-optimize-mundlejs

Introduction

Mundlejs is a dynamic, client-side javascript bundler used to deliver optimized bundles of javascript to large, modular, browser-based apps. Check out the github repository and nodejs discussion board for more description of what it does and why.
Although it is dynamically delivering efficient, optimized bundles of javascript, mundlejs's basic competition is static file servers that deliver large concatenated javascript files with an entire application loaded. Therefore, while I don't expect to compete with nginx, apache, etc. in performance, it is necessary to at least have performance be of the same order of magnitude, so the advantages of mundlejs can actually be relevant, not vastly overridden by disadvantages.
The following is a description of my benchmarking setup and the baseline results I achieved with a completely un-optomized mundlejs. At this stage, mundlejs is basically not much more than a proof-of-concept and api demo with numerous glaring potential areas of improvement, so don't be put off by the extremely poor results. What I'm attempting here is a baseline that I will compare to as I go forward, and it's something that, at this stage, should be quite easy to massively improve upon.

Setup

Machine

Testing was performed on my main development machine, a 27-inch, Mid 2011 iMac.

• Processor: 3.4 GHz Intel Core i7 (4 physical cores with hyper-threading, 8 cores)
• Memory: 16 GB 1333 MHz DDR3
• OS: Mac OS X Lion 10.7.4 (11E53)

I did not bother to reboot before each benchmark or ensure that there were not unnecessary other processes running, so the system had my base computing load of Mail, Safari, Chrome, TextEdit, Skype, oDesk Team, iTunes, etc.

Both server and the benchmarking client were run on the same, single machine.

These shortcuts are appropriate because I'm doing comparative testing at the scale of orders of magnitude. If/when mundlejs gets closer in performance to the competition, I'll consider tightening up my benchmarking setup.

Client

The client setup I used is the ab.c wrapper around ApacheBench, HTTPerf, and Weighttp. I followed the author's recommendation to use Weighttp (at this stage, it really doesn't make much difference for me). In order to get it working on my Mac OS X machine I needed to massage the wrapper ever so slightly. If I'm not forgetting anything, this involved three things:

• Comment out the dependency on linux/major.h (I just tried this and it worked, didn't dig too deeply into why it was there in the first place)
• Manually set the number of processor cores in my machine, because OS X doesn't have /proc/cpuinfo
• Install libev in order to be able to build and install weighttp
• In order to increase the number of available file descriptors to 200000, I needed to change the system limits, kern.maxfiles and kern.maxfilesperproc

For some reason, at a concurrency level 140 while testing Mundlejs and 170 while testing Apache, I got Connection Reset by Peer (54) errors. Rather than take the time to dig into them, I just reduced the upper limit of concurrency being tested to 100 (instead of 1000).

If you run into other problems, it's probably something I forgot, let me know and I'll try to remember how I worked around them.
I lowered the number of requests to perform at each concurrency level to 1000 (from 1000000) to speed up the tests (mundlejs is excruciatingly slow ;) Surprise!). Maybe when I'm doing more fine tuning later it might be necessary to have more statistical precision but, at this point, with orders of magnitude of difference already evident its really not necessary to go crazy about getting very precise average results.

Server

I tested three servers, Mundlejs, the default installation of apache on OS X, and the nodejs connect library.

The javascript that I served is the codebase of koding.com circa April 2012. I used it because it is about 1.3MB of javascript spread across 133 files, with all kinds of real-life interdependencies. As I obviously can't make their codebase open source for them ;) I haven't included the test code in the repository. So, if you want to repeat these tests, you'd have to get your own test files, which I'm sure could be done in some clever automated way.

For Mundlejs, ab.c uses the Mundlejs api to request a single file, which has dependencies on other files, which have dependencies on other files, etc. until the entire 1.3MB gets rolled into one bundle and served back to the client.

For apache and connect, I concatenated all the javascript files together into a single 1.3MB javascript file. In the case of apache, I just placed that in my ~/Sites folder, turned on web sharing, and pointed ab.c at the appropriate address. For connect, I used connect.static, set the base directory to the appropriate folder and pointed ab.c at the appropriate address.

Mundlejs

These benchmarks were run with the 86c38f218b44af63894c6cea81e05257f0ef98bc commit of the master branch of mundlejs.

Results

The results for the three different servers are clearly distinct, the top group is Apache, the second Connect, and the slowest is mundlejs.

In the coming weeks, I will begin optimizing mundlejs and then rerunning these benchmarks to see the effects of my optimizations.

Update:

Turns out node.js 0.8.0 was officially released yesterday. So I reran the benchmarks after upgrading and there was an improvement, although it was not significant enough to really show on the scale of the above chart. Comparing mundlejs on node.js 0.6.19 and 0.8.0: