giant dancing chipmunk

Having managed systems for large (>1m users) web applications I had to watch the podcast of Chris Messina and Larry Halff discuss lessons learned from the recent catastrophe which took down Ma.gnolia. In a nutshell, Ma.gnolia didn’t have good backups and the worst case scenario took place which led to massive data loss. I would like to give Larry kudos for upfront transparency and facing his users; I’m sure dealing with the problem was stress enough. I feel Larry’s earned a red badge of courage regardless of outcome.

“If you’re a startup, don’t do your own IT at all.”
-Larry Halff

This quote is Larry’s “overall lesson learned” and for the sake of other web applications I’m going to take this out of context and completely disagree. As cool as Ma.gnolia was and regardless of where it may go, this is the wrong lesson and in fact, it’s bad advice and presents problems which got Ma.gnolia into this situation in the first place.

The real lesson here is that IT should not be underestimated. Business depends on IT and it’s necessities are as dire as employees’. Unfortunately it’s easy to overlook IT and when you do there’s rarely evidence until it’s too late. Not paying attention to IT is like running red lights and stop signs — one day you’re going to get hit and it’s going to hurt. Outsourcing just makes you a passenger without a seat belt. If you’re a startup, especially a technology startup, you need to be in the driver’s seat.

At one point in time this mostly applied to technology-based companies but that time has passed for a few reasons. First, IT is flooded with service providers who suck and service providers rarely admit liability. Second, companies with cores businesses outside of technology are no longer able to remain competitive without including technology in their core business processes. Five years ago you could outsource your IT if your business revolved around paperwork or physical labor but if you’re doing so now you’re facing significant costs and risk versus training or recruiting in-house personnel. I would post numbers to back myself up for these costs but anyone can find numbers for their cause.

I’m not saying that outsourcing isn’t the right thing to do, in fact, it can be a competitive advantage; but you can’t outsource the responsibility bound to your IT systems. Fortunately Ma.gnolia can rebuild a better service with no capital, limited effort, and no defense lawyers. In doing so, their IT systems should be treated with as much priority as core operations and as much scrutiny as financials. Your CPA isn’t responsible for your tax errors and your service provider won’t be responsible if their backup system fails (it happens more than you think!). If you’re going to outsource IT then know what you’re getting into, maintain your options on a monthly or quarterly basis, have a backup plan, and don’t put all your eggs in one basket. This time Ma.gnolia has the responsibility of not fucking up their reputation a second time. If I were Ma.gnolia I wouldn’t want to outsource the rest of my dignity. If I did, I would be very scrupulous.

In the area of web applications there a few prominent service providers I know of which could be considered if you’re going to have someone else do your IT. Those are Engine Yard, Rails Machine, and RightScale. YMMV and you must keep doing your homework on service providers because while they maybe good today they could be gone tomorrow. Another option, which Larry is taking and should be applauded, is to open source the effort and involve your community in the process. Twitter has gone through a similar affair and I’m sure the service is better for it and am confident Ma.gnolia will also see positive results.

I don’t feel as though Larry is headed in the wrong direction or that I can speak for Ma.gnolia but if you’re a startup you should do your own IT – just don’t do it alone.

As an aside I’d also like to point out a subject that’s important and should be considered no matter what IT solutions are chosen. Monitoring, gives you a better chance at resolving issues before they become critical as well as a constant perspective of operations. Without monitoring you might not know if backups succeed, if your hard disks are healthy, or if you’re running out of disk space. Moreover, monitoring tells everyone involved what systems are crucial as well as providing history for analysis. When considering IT systems don’t forget the proactive value of monitoring.