CloudPundit: Massive-Scale Computing

Something that I’ve been thinking about: The shift to global-class computing, and massively scalable infrastructure, represents a fundamental shift in the skill sets that will be valued in IT Operations.

Those of you who, like myself, have worked at service providers in hyper-growth mode, are already familiar with what occurs when you need to grow at red-shift speeds: You automate everything humanly possible, and you try to standardize the heck out of things. Usually you end up trying to make sure that your infrastructure is horizontally scalable, and that your hardware is as interchangeable as possible, alllowing any single server to fail and the system as a whole to go chugging along, while you eventually go yank that server out and replace it with another just-as-generic box that you’ve auto-provisioned.

The shift to the cloud model, whether public or private, basically pushes the idea that every IT organization does that, either in-house or through the services of a provider. It puts the premium on software development / scripting skills — these are the guys who automate things and who write the glue for integrating your toolsets. You’ll have a handful of guys who are your serious architects — the guys who tune and optimize your hardware and storage, design your configurations, and so on. (That might be a single guru, or you might go to consultants for that, alternatively.) You’ll have a few folks who know the operational ins-and-outs of troubleshooting your applications. Everyone else becomes a hardware monkey, entry-level folks who don’t need much more of a skillset than it takes to assemble a PC from parts.

This is writ large in the Google model of Operations, but it’s been true for the last decade in every dot-com of significant size, too. Your hardware operations guys are rack-and-stack types. Everyone else blends systems administration with scripting abilities, and because your toolsets have to scale and be highly maintainable, this is scripting that has the air of serious development, not a one-time thing that can be banged out unreadably.

The routine drudgery of IT Operations is going to get automated away, bit by bit. Right now, many enterprises still operate at a scale and lack of standardization that means that it’s not necessarily more efficient to automate a task than simply do it manually. In the cloud model, the balance tips to the automation side, and the basic value of “I can wrangle boxes” declines precipitously.

My advice to sysadmins: If you are not fluent in a scripting language, and/or not capable of writing structured, readable, maintainable, non-hackish code, now is the time to learn.