Catalyst on JustHost with fastcgi

Update- Blog moved to: http://saleemabdulhamid.com/blog/2012/10/catalyst-on-justhost-with-fastcgi

I'm working on upgrading my personal site (and will also be consolidating this blog there) and have decided to go with the Catalyst web framework for Perl. I have the site hosted on JustHost, with shared hosting (no dedicated ip) so my options were limited to php, perl and ruby on rails.

I decided to go with Perl for a number of reasons, one of them being that I've been personally interested in the language for a while. Another is that I read a lot of comparisons that pointed towards rails being easier to get started but harder to extend and modify outside of the mold of how things are normally done. I'm not sure about the truth of that.

Anyway, it seems that JustHost's Perl support is more geared towards simple cgi scripts than modern Perl and Catalyst, so it was a bit difficult to figure out the appropriate setup to run Catalyst on there. So here is what you have to do, and I imagine this would be representative of a lot of other cheap shared hosting providers. The biggest issue is that you can't run the Catalyst server because you can't open a firewall port, so if you run it, it won't be visible to the outside world. You also can't use cgi, because it has serious performance issues when running an application like Catalyst.

The .htaccess file

JustHost uses an apache server, running behind the scenes and serving all the sites on the shared server. You need to create a new (or modify the existing) .htaccess file to tell the apache server what to do. The .htaccess file will be at the root of whatever directory your domain points to. If you only host a single site on the server, that will probably be in the public_html folder. If you host multiple sites, check what folder the subdomain points to (usually you register multiple domains with JustHost and then alias them to subdomains on your main domain; those subdomains point to different folders in your server).

In your .htaccess file, first you need this line:

AddHandler fcgid-script .pl

This tells apache to use fastcgi to handle files that have the .pl extension.

The second thing you need to do is redirect requests to your domain to the script, which is located somewhere inside your Catalyst project directory:

RewriteEngine On
RewriteRule ^.*$ myapp/script/myapp_fastcgi.pl [NC,L]

This redirects all  requests to the Catalyst automatically generated fastcgi script inside the script folder of your project.

Including your CPAN modules

There's one additional thing needed, in order to be able to access the installed cpan modules from your application. You use JustHost's cPanel to install Perl modules. Then you need to add this to the myapp_fastcgi.pl script so the script can find those installed modules (if you need to use any of the other automatically generated catalyst scripts, such as _create.pl, you'll have to add this there, too):

BEGIN {
    my $b__dir = (-d '/home3/detaile7/perl'?'/home3/detaile7/perl':( getpwuid($>) )[7].'/perl');
    unshift @INC,$b__dir.'5/lib/perl5',$b__dir.'5/lib/perl5/x86_64-linux-thread-multi',map { $b__dir . $_ } @INC;
}

This is actually taken from JustHost's documentation and just adds the path to the installed modules. This should be right up at the top of the file, just below the shebang.

Teenagers

Update- Blog moved to: http://saleemabdulhamid.com/blog/2012/9/teenagers

http://www.alexchediak.com/2011/02/teenage-rebellion-not-a-universal-phenomenon.php

Regarding Islam's Position on Gandhi's Nonviolent Resistance

Update- Blog moved to: http://saleemabdulhamid.com/blog/2012/8/regarding-islams-position-on-gandhis-nonviolent-resistance

This article, published in 1939, is part of an addendum to the book I'la as-Sunan, by Mawlana Zafar Ahmad 'Uthmaani. The addendum is titled: "Refutation of Composite Nationalism."

A deceiver (dajjal) from the idolaters arose in India who the advocates of composite nationalism from the members of the Congress take as their leader and guide, by his judgement they come and go, and his instruction in politics they carry out. The accursed claims that resistance against the oppressive rulers by tolerating the persecution is preferable to war, so whoever wishes to throw the noose of the authority of the [oppressive] rulers from his neck, he must oppose the state laws and he must endure what he suffers from the ruler and his oppressive supporters of physical torture and imprisonment; they should do so for a period of time, and by this the pillars of the government will crumble, and its terror and its awe will be removed from the hearts by lengthening the tongues [against them]. We do not deny this being a scheme from the schemes of war and a tactic from its tactics which may occur during a time when its people are too weak to resist the enemy by force, courage, striking and wounding. However, it is very strange that a group of Muslims and their scholars in India have claimed this to be a Sunnah from the examples of the Prophet (Allah bless him and grant him peace) and that he would use it when residing in Makkah before the migration to Madinah, so he commanded those with him of the Muslims to declare the oneness [of Allah] and insult the gods of the idolaters and make mockery of their way of life and criticise their religion and insult their forefathers, and when they suffered persecution from the idolaters, they should not repay them with its like, rather they should pardon and overlook, and endure what they suffer, and this [equates to] resisting the enemy by perseverance and nonviolence. We say: If that was a Sunnah from the traditions of the Prophet (Allah bless him and grant him peace) you would have been the first to call to it and the foremost of those who bowed to it, and you would not have been beaten to it by this accursed, and he would have been following you in this, and you would not be following him; for, has the Sunnah of your Prophet been hidden to you until this accursed explained it or did you not reflect on the Qur‟an, neither you nor your forefathers, and that pitiful one understood it? And if tolerating persecution was always better than war as this leader of yours claims, why did the Prophet (Allah bless him and grant him peace) migrate to Madinah and did not eliminate the strength of the people of Makkah by forgiveness and pardon and nonviolence his entire life? And why did Allah permit him to fight the enemy and commanded him to prepare whatever he could of power, of trained horses, by which the enemy of Allah and his enemy and others besides them would be terrified (Qur‟an 8:60)?

If you say that was only after attaining power in Madinah and when it is so, fighting is preferable, but before attaining it, nonviolent resistance is superior, we say: This conflicts with the opinion of your accursed leader, for he claims nonviolent resistance is always better which is in conflict with what the Qur‟an and the Sunnah of the Prophet (upon him blessing and peace) has come with so it is not permissible for you to support his view at all, nor adduce proof for it from the Qur‟an and Sunnah foolishly and ignorantly. 

From the catastrophes [of this time] is that the accursed has arranged a syllabus for children‟s education and included in this [syllabus] his view that nonviolence is always better than violence and conquering lands with gentleness and softness is better than shedding the blood of people, and his likeness in this is not but as the likeness of the learned men of the Christians who, when they emerge from their lands to spread Christianity, they command the arms factories to prepare weapons of war of heavy artilleries and other [weapons] every day, and when they come to the land of India and other [lands] they announce the saying of the Messiah, “Verily the land of God and His Kingdom belong to the weak and the downtrodden.” Similar [to them] is this accursed who has ordered the elders of his people to build a college of war, so the children of the idolaters learn therein the sciences of war and its arts, while he says to people that nonviolence is always superior to violence. It is strange from the common Muslims and a group of their scholars that they were deluded by his speech and they find no fault in teaching these myths, and they don‟t know that when their children learn this and study Brahman history whose religion was nonviolence, the Brahmans will become great in their hearts, and the greatness of „Umar the Distinguisher, and „Ali and Khalid (Allah be pleased with them) will be removed from their hearts. The Satan beautified in their hearts the superiority of the religion of the Brahmans over the religion of Islam due to their belief that the first is premised on softness, leniency and mercy, while Islam is premised on jihad and shedding the bloods of the idolaters, while they are heedless of mercy to the mischief makers being a great injustice to all the worlds, especially the weak. 

The truth is that gentleness, leniency, forgiveness, pardoning and nonviolence are wonderful and beautiful in its right place; however, when a man is a mischief maker possessing great evil and baseness, with no deterrent preventing him from his evil, he will only grow in pride and rebellion by gentleness and leniency, and [will only grow] in arrogance and violence by forgiveness and pardon. Hence, gentleness with [someone] like him is like showing compassion to snakes and scorpions, which a sane person will never approve, and a possessor of intelligence will never be satisfied with it. Hence, Islam is the religion of beauty (jamal) and majesty (jalal), and [the religion of] those who are harsh to the disbelievers and merciful amongst themselves (Qur‟an 48:29). 

After this, know that the order of Allah to His Prophet (Allah bless him and grant him peace) to forgive, pardon and turn away from the idolaters before the migration was not because this was from the means of resisting the enemy, nor was it because it was something that will eliminate its power and destroy its structure, and if that were so, He would never have ordered them to fight and He would not have said: “We sent down iron in which there is strong power, and benefits for the people” (57:25). Rather, this was only because – and Allah knows best – they were weak in Makkah, having no power to fight, for fear that the people will snatch them. Then when He gave them shelter in Madinah and supported them with His help so the people of Madinah became their supporters and helpers, and He sustained them with wholesome [provisions], He commanded them to resist the enemy by war and fighting. In sum, in Islam there is only fighting with weapons, or not fighting with it. 

As for war without weapons, i.e. by patience and enduring persecution, there is no trace of this in the example of the Prophet (Allah bless him and grant him peace) and the conduct of his companions. Thus, his conduct in Makkah was to avoid fighting and warring altogether, not warring by means of patience as they claim. And his conduct in Madinah was to fight and combat with swords. This does not entail that resistance by means of patience is not permissible in the Shari„ah. It means only that it is not a Sunnah and is not established from the predecessors, so its ruling is the ruling of newly-invented modern weapons – there is no harm in their use when free from something forbidden in the Shari„ah. For if we were to know that the authorities of the time are frightened by merely shouting at them and the foundations of their kingdom will tremble by the peoples demonstrating against them by mere raids and clamouring without fighting and conflict, what need is there for us to throw ourselves into destruction and fight with weapons? For indeed he (Allah bless him and grant him peace) said: “Do not wish to meet the enemy, and ask Allah for safety. And when you meet [them], stay firm.” (Narrated by al-Bukhari and Muslim) However, the permissibility does not entail superiority, nor [does it entail] fighting with weapons is always blameworthy as the accursed claims. Abu al-Zubayr narrated from Jabir that “the Prophet (Allah bless him and grant him peace) remained for ten years following people into their homes in the season [of Hajj] and [the festivals of] Majannah and „Ukaz, [saying]: „Who will believe me and who will shelter me and who will help me until I convey the messages of my Lord, and for him is Paradise?‟ And he did not find anyone to help him and shelter him, and he walked amongst their men inviting them to Allah while they pointed to him with the fingers, until Allah sent us [the Ansar] to him from Yathrib, so a man from us would come to him and believe in him and read to him the Qur‟an, and then he would return to his family and they would accept Islam, until there did not remain a house from the houses of the Ansar except there was a group of Muslims therein openly declaring Islam. Allah sent us to him so we conferred and convened and we said: „Until when will the Messenger of Allah (Allah bless him and grant him peace) be driven to the mountains of Makkah and feared for?‟ So we travelled until we arrived to [meet] him in the season [of Hajj] and we pledged the pledge of „Aqabah...” (Zad al-Ma„ad, 1:306) 

Whoever contemplates the conduct of the Prophet (Allah bless him and grant him peace) and his companions during their stay in Makkah, he will not doubt that the weak from them would endure and forgive and pardon those who oppressed them. As for the strong from them, they would defend and fight. Ibn Ishaq said: “The companions of the Messenger of Allah (Allah bless him and grant him peace) would, when they prayed, go to the mountain paths and shorten their prayer [for fear] of their people. When Sa„d ibn Abi Waqqas was amongst a group of the companions of Allah‟s Messenger (Allah bless him and grant him peace) in a mountain path from the mountain paths of Makkah, suddenly a group of idolaters emerged while they prayed and they abused them for what they were upon and [what they were] doing, until they fought them. So Sa„d ibn Abi Waqqas on that day struck a man with the jawbone of a camel and he wounded him. That was the first blood to be spilt for Islam.” (Sirah Ibn Hisham, 1:148) It is authentic that „Umar ibn al-Khattab, when he became a Muslim, he fought Quraysh and they fought him until he prayed openly near the Ka„bah and the Muslims prayed with him. Ibn Mas„ud said: “When „Umar became Muslim, the Muslims became dignified in themselves, together with Hamza‟s acceptance of Islam before him. And they knew that they would protect the Messenger of Allah (Allah bless him and grant him peace) and they will take revenge by means of them from their enemy.” (Sirah Ibn Hisham, 1:188) 

So if endurance of persecution and forgiveness and pardoning was from the means of resistance, Sa„d would not have fought any of the idolaters, nor „Umar, nor Hamzah. And those who were tortured from the Muslims like Bilal, „Ammar ibn Yasir, his mother and his father and their likes from the slaves, were only tortured because they were weak and unable to protect themselves, and their endurance of persecution was not a scheme or a war tactic, and nor was it a means of resisting the enemy, rather it was because of the lack of means and strength. And when the Messenger of Allah (Allah bless him and grant him peace) saw what had afflicted his companions of suffering and [saw] the safety he was in from Allah and from his uncle Abu Talib, and that he was unable to protect them from what they suffered, he said to them: “If only you would leave for the land of Abyssinia, for there is a king there under whose [authority] none are oppressed and it is a land of integrity, until Allah makes an opening for you in [the plight] that you are in,”; thereupon, Muslims from his (Allah bless him and grant him peace) companions went to Abyssinia (Sirah Ibn Hisham, 1:172). 

Thus, if the endurance of the Muslims, their forgiveness and pardoning of the idolaters was a scheme from the schemes of war, there would be no meaning to their migration to Abyssinia. When Abu Talib died and Quraysh began to persecute the Messenger of Allah (Allah bless him and grant him peace) more than ever before in the life of his uncle, he went to Ta‟if and he exposed himself to the tribes, so Allah sent to him the people of Madinah as was previously mentioned, and Allah permitted His Messenger to fight and defend and take revenge from those who oppressed them and transgressed against them; thus, the first verse that was revealed on this is His statement: “Permission (to fight) is given to those against whom fighting is launched, because they have been wronged, and Allah is powerful to give them victory.” (22:39) Ibn Ishaq said: “When Allah Almighty permitted him to fight, and this group of the Ansar followed him to help him and [to help] those following him and those Muslims who sought refuge with them, the Messenger of Allah (Allah bless him and grant him peace) ordered his companions from the emigrants of his people and those with him in Makkah to leave for Madinah and migrate there and join their brothers of the Ansar and he said: „Verily Allah has made for you brothers and an abode you will be safe in,‟ so they left gradually. And the Messenger of Allah (Allah bless him and grant him peace) stayed in Makkah waiting for permission from his Lord to leave from Makkah and migrate to Madinah.” (Sirah Ibn Hisham, 1:257) 

Hence, if his endurance of persecution and forgiveness and pardon of the enemy was due to it being a form of resistance, a scheme from the schemes of war, none of this would have meaning as is not hidden. And if his endurance and the endurance of those with him of the Muslims was to seek kingdom and authority, as these fools from the Congress claim, the Messenger of Allah (Allah bless him and grant him peace) would not have rejected „Utbah ibn Rabiah‟s offer when he said to him: “O my nephew, you are one of us as you know, of the noblest of the tribe and hold a worthy position in ancestry. You have come to your people with an important matter, dividing the community thereby and ridiculing their customs, and you have insulted their gods and their religion, and declared that their forefathers were unbelievers, so listen to me and I will make some suggestions, and perhaps you will be able to accept one of them.” The Messenger of Allah (Allah bless him and grant him peace) agreed, and he went on, “If what you want is money, we will gather for you out of our property so that you may be the richest of us; if you want honour, we will make you our chief so that no one can decide anything apart from you; if you want sovereignty, we will make you king, and if this ghost which comes to you, which you see, is such that you cannot get rid of him, we will find a physician for you, and exhaust our means in getting you cured of it,” or words to that effect. The Messenger of Allah (Allah bless him and grant him peace) listened patiently, and then said: “Now listen to me, „In the Name of Allah, the compassionate and merciful, ha mim, a revelation from the Compassionate, the Merciful, a book whose verses are expounded as an Arabic Quran for a people who understand, as an announcement and warning, though most of them turn aside not listening and say, Our hearts are veiled from that to which you invite us.‟ (Qur‟an 41:1-5).” Then the Messenger of Allah (Allah bless him and grant him peace) continued to recite it to him. When „Utbah heard it from him, he listened attentively, putting his hands behind his back and leaning on them as he listened. Then the Messenger of Allah (Allah bless him and grant him peace) ended at the prostration and prostrated himself, and said, “You have heard what you have heard, Abu al-Walid; the rest remains with you.” (Sirah Ibn Hisham, 1:155) [Translation extracted from Alfred Guillaume‟s The life of Muhammad, pp. 132-3] 

Thus, can anyone with a grasp of intellect say the Messenger of Allah (Allah bless him and grant him peace) would endure the persecution of Quraysh and forgive them and pardon in order to seek leadership and to eliminate their supremacy or he would do that to resist the enemy by endurance and nonviolence, whereas they would offer to him leadership willingly with his authority over them if he were to abandon dividing their community and ridiculing their customs and insulting their gods and their religion, and they never requested from him to leave his religion for their religion, nor to leave calling people to his religion, and they only asked from him to abandon dividing the community by insulting their religion and ridiculing their customs and attributing error to their forefathers? Ibn Ishaq said: “Then when the Messenger of Allah (Allah bless him and grant him peace) openly declared Islam to his people and proclaimed it as Allah commanded him to (15:94), his people did not disown him and they did not reject him according to what has reached me, until he mentioned their gods and criticised them; and when he did this they deemed him a great [threat] and denounced him and resolved to oppose him and show hostility to him, except those that Allah protected by [making them] accept Islam, and they were few.” (Sirah Ibn Hisham, 1:138) 

You may [even rightly] say that the people were offering him obedience and were satisfied with him being a ruler over them had they made an agreement of composite nationalism between him and them whereby no one interferes with another in the matter of his religion, he does not insult him, accuse him of error, insult his religion, and all people are equal in the freedom of consciousness and belief, so the Messenger of Allah (Allah bless him and grant him peace) rejected this and refused but to complete his light even if the disbelievers disliked it (Qur‟an 61:8). In all of this is the greatest proof of the invalidity of composite nationalism which the fools of our times claim is established by the Qur‟an and Sunnah, and is the clearest proof that the endurance of the Prophet (Allah bless him and grant him peace) of the persecution of his people and his forgiveness and pardoning them was not in order to seek leadership, nor to break the supremacy of the enemy thereby, rather to accomplish the proof and to spread the message and prove that he does not desire the gains of the world but only wants rectification [of people] as much as is possible (Qur‟an 11:88). Such was the pattern of Allah with His Prophets that they were tried with the persecution of their peoples firstly, then they were patient, then Allah decided between them and their peoples and He is the best of deciders. Do you not see Musa (upon him blessing and peace) and his people, how Pharaoh and his people persecuted them? They afflicted them with a horrible torment, killing their sons and sparing their women (Qur‟an 2:49) so they endured this for a while, and their endurance was not a resistance of the enemy, rather in obedience to the command of Allah that He may try them, and until the trial was complete and the tribulation was widespread, Allah caused Pharaoh and his people to drown, and His “sublime word was fulfilled for the Children of Israel, because they stood patient” (7:137) “and We destroyed what Pharaoh and his people used to build and what they used to raise high.” (7:137) Thus, whoever claims that the endurance of Musa and his people was a part of nonviolent resistance, he has caught himself in ignorance. If that were so, Allah would not have drowned Pharaoh and his people; rather, He would have commanded the Children of Israel to endure and always forgive until their patience destroyed the kingdom of Pharaoh and shook its foundations and destroyed its structure. Since it was not so, analogise this to the endurance of the Prophet (Allah bless him and grant him peace) and his companions; for al-Bukhari narrated from Khabbab ibn al-Arat, he said: We complained to the Prophet (Allah bless him and grant him peace) while he was reclining on a garment in the shade of the Ka„bah and we faced harshness from the idolaters, saying: “Will you not pray to Allah?” So he sat up with his face having turned red, and he said: “There was a man from those before you, for whom a [ditch] was dug in the earth, and he would be placed therein, and a saw would be brought and placed above his head and he was split in two, and that did not deter him from his religion; and [another] was raked with iron combs that removed his flesh from his bones and sinews, and that did not deter him from his religion. By Allah! Verily, this matter [of Islam] will be fulfilled, until a rider travels from San„a to Hadramawt not fearing [any] besides Allah, or a wolf [devouring] his sheep, but you haste.” (Mishkat al-Masabih, pp. 447) 

Do you, then, believe that this endurance and tolerance of suffering was a desire for leadership? Never! Rather, it was so Allah exposes the patient (Qur‟an 3:142), and distinguishes between the truthful and the liars (Qur‟an 69:3), and then permits them to fight (Qur‟an 22:39), suppressing thereby the disbelievers (Qur‟an 3:141). So understand. Allah has authority over your guidance and He is the protector of the righteous. 

Let this be the last sentence on the issues circulating on the tongues of the commoners in these days. To Allah is the complaint of what those affiliated with knowledge are perpetrating, of the distortion of texts and laws.

[The Author‟s Dream of the Prophet (peace and blessings be upon him)]

I was blessed in [my] sleep to see the master of creation, our master Muhammad (Allah bless him and grant him peace), in these days, and he gave me glad tidings of the dominance of Islam and its people over the idolaters and the blameworthy disbelievers soon with the help of Allah, Possessor of Glory and Generosity. O Allah! Make this dream of mine true. Provide us with complete and true sincerity, and give us the ability to [do] what you love and approve, and give us and those who follow us a good end, and make our latter better than our former. Pardon us, forgive us, and have mercy on us. You are our patron, so help us against the disbelieving people.

[Final Remark]

A draft of this addendum was completed towards the end of Dhu al-Qa'dah in the year 1357 after the migration of our Prophet, upon him blessing and peace, a perpetual, lasting and continuous blessing and peace, without end (January 1939 CE). All praise to Allah by Whose glory, greatness and good favour, righteous deeds are accomplished. 

Source: http://ukkhuddam.files.wordpress.com/2011/11/refutation-of-composite-nationalism-mawlana-zafar-ahmad-al-uthmani.pdf

Big Improvements!

Update- Blog moved to: http://saleemabdulhamid.com/blog/2012/7/big-improvements

Introduction

Another day, another (two) optimizations- one of which turned out to be a really big improvement.

Previous posts in this series:

• Baseline
• First Optimization

Cacheing the File Contents

The first optimization has to do with reading-in of files from the file system. Unlike a traditional static file server, we need the contents as a string in javascript memory (as opposed to being stored as a buffer) for two reasons- (a) we need to parse files for dependencies (which could be done before the server starts listening, since we cache those dependencies) and (b) we need to dynamically join multiple files into optimized bundles. In the non-optomized version of mundlejs, a file is read from the file system every time it is requested, or when a file is requested that has it as a dependency. As of this commit, the file contents are kept in memory so they only need to be read from the file system the first time that particular file is needed. In the future, as part of a deploy-mode, we could go ahead and get all the file contents in memory before the server starts listening. Here are the results with this optimization:

Again we have a definite improvement, but still nothing in the same order of magnitude as our competitors.

Cacheing the Bundled Javascript in Buffers

The last very obvious, low-hanging-fruit-with-a-lot-of-promise optimization has two components.

Although bundling of javascript happens dynamically, for a given file request and list of already loaded dependencies, the bundle that returns will always be the same. Depending on the different workflows a user might take through a client-side application there is usually a pretty high chance of multiple clients requesting the same bundle in this fashion. So it's not necessary compare the already-loaded files to the dependencies of the requested files and build the bundle from scratch every time- we can just cache that bundle the first time it is created. Since we are already cacheing the dependency list and the file contents, this is probably not a really big optimization, but it should still have some positive effect in terms of cutting down the amount of computation spent per request.

Cacheing these bundles leads to an opportunity to optimize the biggest bottleneck we have- because the server response needs to be built up dynamically, we build it as a string in memory. When it is served, V8 has to make a copy of the string- because the address of javascript variables in memory can change at runtime, the data can't be sent directly to the network. This operation is very expensive and is incurred for every request. The solution is, since we're already cacheing the bundles, instead of cacheing them as strings in memory, convert them to buffers and cache those. Buffers point to real memory locations outside of the V8 heap and therefore can be sent directly to the network on every subsequent request for that particular bundle.

Here are the results of this optimization:

Suddenly, we're on the same order of magnitude performance as Apache and Connect, and in fact seem to be beating Connect!
Now that we have performance in the same ballpark as these others, I will try and rerun the benchmarks on a higher concurrency range, with a larger number of iterations, and on a clean reboot for each run. Perhaps, I'll also run some other popular servers and also try to optimize some settings of the other servers to get more accurate results.

Next Steps

I'll probably take a little break from optimizations now to focus on perfecting the testing coverage, especially of the client-side code which is only covered by manual tests at this time.

Optimizing Mundlejs: Cacheing the dependency parse

Update- Blog moved to: http://saleemabdulhamid.com/blog/2012/7/optimizing-mundlejs-cacheing-the-dependency-parse

Introduction

Following my previous efforts at establishing a baseline benchmark for mundlejs, I took my first step at optimization. Previously, the contents of every javascript file requested were parsed for dependencies. This occurred on every request from a different client. As of this commit, the direct dependencies of each file are cached, so the next time that file is requested, it is not necessary to parse the file again. This is a precursor to being able to pre-parse all files prior to beginning to serve files, which will be one of the features of the "deploy mode." Because we don't have cluster support built in yet, in the benchmarks below all parses are done 8 times (or however many cores are on the testing machine), since the cache is not shared between worker processes yet.

Results

We're still not on the same order of magnitude as Apache and Connect, but there was a significant improvement. The little dip at one data point is probably just an artifact of something else that was happening on my machine at the same time. I don't think I'll be exploring it further, preferring to spend my time on the next step in optomization.

Preparing to optimize Mundlejs

Update- Blog moved to: http://saleemabdulhamid.com/blog/2012/6/preparing-to-optimize-mundlejs

Introduction

Mundlejs is a dynamic, client-side javascript bundler used to deliver optimized bundles of javascript to large, modular, browser-based apps. Check out the github repository and nodejs discussion board for more description of what it does and why.
Although it is dynamically delivering efficient, optimized bundles of javascript, mundlejs's basic competition is static file servers that deliver large concatenated javascript files with an entire application loaded. Therefore, while I don't expect to compete with nginx, apache, etc. in performance, it is necessary to at least have performance be of the same order of magnitude, so the advantages of mundlejs can actually be relevant, not vastly overridden by disadvantages.
The following is a description of my benchmarking setup and the baseline results I achieved with a completely un-optomized mundlejs. At this stage, mundlejs is basically not much more than a proof-of-concept and api demo with numerous glaring potential areas of improvement, so don't be put off by the extremely poor results. What I'm attempting here is a baseline that I will compare to as I go forward, and it's something that, at this stage, should be quite easy to massively improve upon.

Setup

Machine

Testing was performed on my main development machine, a 27-inch, Mid 2011 iMac.

• Processor: 3.4 GHz Intel Core i7 (4 physical cores with hyper-threading, 8 cores)
• Memory: 16 GB 1333 MHz DDR3
• OS: Mac OS X Lion 10.7.4 (11E53)

I did not bother to reboot before each benchmark or ensure that there were not unnecessary other processes running, so the system had my base computing load of Mail, Safari, Chrome, TextEdit, Skype, oDesk Team, iTunes, etc.

Both server and the benchmarking client were run on the same, single machine.

These shortcuts are appropriate because I'm doing comparative testing at the scale of orders of magnitude. If/when mundlejs gets closer in performance to the competition, I'll consider tightening up my benchmarking setup.

Client

The client setup I used is the ab.c wrapper around ApacheBench, HTTPerf, and Weighttp. I followed the author's recommendation to use Weighttp (at this stage, it really doesn't make much difference for me). In order to get it working on my Mac OS X machine I needed to massage the wrapper ever so slightly. If I'm not forgetting anything, this involved three things:

• Comment out the dependency on linux/major.h (I just tried this and it worked, didn't dig too deeply into why it was there in the first place)
• Manually set the number of processor cores in my machine, because OS X doesn't have /proc/cpuinfo
• Install libev in order to be able to build and install weighttp
• In order to increase the number of available file descriptors to 200000, I needed to change the system limits, kern.maxfiles and kern.maxfilesperproc

For some reason, at a concurrency level 140 while testing Mundlejs and 170 while testing Apache, I got Connection Reset by Peer (54) errors. Rather than take the time to dig into them, I just reduced the upper limit of concurrency being tested to 100 (instead of 1000).

If you run into other problems, it's probably something I forgot, let me know and I'll try to remember how I worked around them.
I lowered the number of requests to perform at each concurrency level to 1000 (from 1000000) to speed up the tests (mundlejs is excruciatingly slow ;) Surprise!). Maybe when I'm doing more fine tuning later it might be necessary to have more statistical precision but, at this point, with orders of magnitude of difference already evident its really not necessary to go crazy about getting very precise average results.

Server

I tested three servers, Mundlejs, the default installation of apache on OS X, and the nodejs connect library.

The javascript that I served is the codebase of koding.com circa April 2012. I used it because it is about 1.3MB of javascript spread across 133 files, with all kinds of real-life interdependencies. As I obviously can't make their codebase open source for them ;) I haven't included the test code in the repository. So, if you want to repeat these tests, you'd have to get your own test files, which I'm sure could be done in some clever automated way.

For Mundlejs, ab.c uses the Mundlejs api to request a single file, which has dependencies on other files, which have dependencies on other files, etc. until the entire 1.3MB gets rolled into one bundle and served back to the client.

For apache and connect, I concatenated all the javascript files together into a single 1.3MB javascript file. In the case of apache, I just placed that in my ~/Sites folder, turned on web sharing, and pointed ab.c at the appropriate address. For connect, I used connect.static, set the base directory to the appropriate folder and pointed ab.c at the appropriate address.

Mundlejs

These benchmarks were run with the 86c38f218b44af63894c6cea81e05257f0ef98bc commit of the master branch of mundlejs.

Results

The results for the three different servers are clearly distinct, the top group is Apache, the second Connect, and the slowest is mundlejs.

In the coming weeks, I will begin optimizing mundlejs and then rerunning these benchmarks to see the effects of my optimizations.

Update:

Turns out node.js 0.8.0 was officially released yesterday. So I reran the benchmarks after upgrading and there was an improvement, although it was not significant enough to really show on the scale of the above chart. Comparing mundlejs on node.js 0.6.19 and 0.8.0: