I am a hard core believer that innovation almost 100% created from the bottom up. Almost 100% of that innovation in the Web 2.0 space is small companies like Technorati, Six Apart (typepad), Wordpress (the Apache of the Blogspace) and delicious among many others. They work on a small problem, it gets popular, then they have to cope with the challenges of growth.

The big problem this week was Typepad’s 10 hour outage on Friday:

But Typepad failed to inform its users that posts they were making during the downtime period were disappearing into a black hole.
“I spent about eight hours on 12 July updating photo albums and captions for an artist friend of mine. All gone. TypePad, which has had a disturbing history of user complaints and losing people’s work, simply collapsed,” reader Bruce Stidson told us.

Typepad goes titsup again - The Register 

Not only did this problem affect us smaller folk, it also affected blogs from people like Brian Williams from NBC, the Times of London, San Jose Mercury News and others.

What is obvious is that these are very small companies trying to cope with very large problems. Web 2.0 is unique in that you can go from a single user to 5 million inside a year and still be required to have the same reliability that a large multi-billion dollar corporation has.

What is not so obvious is that these companies are required to do it on very small budgets. Even the successful ones have total funding in the range of $10M - $15M, the smaller ones have to do this on much less. This means that their budgets for IT are only in the $1M - $2M range, most of that money is soaked up in salaries and benefits. This does not go very far when you are expected to support millions of users 24 hours per day 7 days a week.

This inevitably means compromise. This means buying servers that are lower cost, and skipping on expensive fail-over and storage solutions because “we can fix that later. Cisco equipment does not come cheap, and they don’t subsidize small companies (most that is). I am not saying that this is any different than any other small company building up, it is that their problem is made more acute by the growth issue.

The biggest capital cost for these types of deployments are networking and server costs. The variable costs are bandwidth and hosted environments, but thankfully a lot of money was invested in the Web 1.0 days where that investment has now been written off.

A significant bonus is the proliferation of free or near free tools such as Linux, BSD, Apache, PHP, MySQL, Ingres, Solaris etc to dramatically reduce the cost of purchasing software tools. In fact don’t underestimate how much Open Source has benefited Web 2.0 Companies. A quick Netcraft reveals the following:

• technorati.com - Linux/Apache
• del.icio.us - Linux/Apache
• Digg.com - Linux/Apache
• wordpress.com - Linux/Litespeed
• typepad.com - Linux/Apache with F5 balancers

Most of these sites are PHP, Python and to a lesser extent Perl. The benefit of these scripting languages are that the knowledge how to scale and cluster these are well know and is freely available.

To get an idea of the scope of the problem challenge, David Silfry posted this on his blog:

• Technorati now tracks over 35.3 37.3 Million blogs
• The blogosphere is doubling in size every 6 months
• It is now over 60 times bigger than it was 3 years ago
• On average, a new weblog is created every second of every day
• 19.4 million bloggers (55%) are still posting 3 months after their blogs are created
• Technorati tracks about 1.2 Million new blog posts each day, about 50,000 per hour

As Web 2.0 and specifically Social Networking sites continue to show these levels of growth, we will expect to continue to see high profile outages, the benefit is however is that a lot of knowledge is building up on how to scale these systems more efficiently.

Technorati Tags: technorati, blog, web+2.0, linux, php, python, perl, typepad, digg, del.icio.us, Internet, cisco, Web, IT