Crisis in the Cloud: How the Tech Bubble Stifles Innovation and Hampers Cloudâ€™s Adoption
The start of 2011 marked the moment that â€œinnovationâ€ arrived as the buzzword on lips of everyone from the president of the Consumer Electronics Association to the president of the United States. Invoked in totemic tones in one speech after another, business leaders have stressed â€œinnovationâ€ (often in contrast to â€œregulationâ€) as the cure for the worldâ€™s economic and cultural ills.
But while innovation enthusiasm was rising among business and government leaders, it also became fashionable in certain quarters to fret that weâ€™re in the midst of an enduring innovation drought. And if my conversations with venture capitalists and entrepreneurs are any indication, then even the cloudâ€”supposedly the cutting edge of technological progressâ€”is facing an uphill battle to innovate. The reason? A combination of easy money and cheap transistors (i.e., Mooreâ€™s Law). Even worse, these factors are also creating a kind of hidden barrier to cloud adoption thatâ€™s potentially larger than the standard â€œbarrierâ€ issues of security, reliability, regulatory compliance, and vendor lock-in.
Innovation and the talent shortage
In January of this year, right as the innovation-speak reached fever pitch, prolific blogger and economist Tyler Cowen launched the â€œinnovation stagnationâ€ conversation into the punditry mainstream with his best-selling Kindle Single, The Great Stagnation . In this weakly argued essay, which serves mainly to demonstrate Paul Krugmanâ€™s point about how little attention high-profile libertarians and other â€œfreshwaterâ€ economist types pay to the opposing teamâ€™s work (Iâ€™m thinking specifically of Marxist economic historian Robert Brennerâ€™s dense, data-rich oeuvre on innovation and â€œthe long downturnâ€), Cowen begins with the observation that technological innovation has essentially flatlined since 1973. He then claims that America has eaten up all of the â€œlow-hanging fruitâ€â€”like free land and a smart yet under-educated populaceâ€”that powered previous waves of innovation.
More recently, Cowenâ€™s arch-libertarian fellow traveler Peter Thiel penned an essay in the National Review that also aimed to illuminate how and why the innovations of the past few decades have failed to measure up to those of the Cold War era and before. And now novelist Neil Stephenson has picked up the baton with an essay entitled â€œInnovation Starvation,â€ which, I kid you not, blames the problem in part on a dearth of good sci-fi.
All of these authors are focused on a perceived lack of fundamental, earth-shaking innovation of the Apollo 11, â€œone giant leap for mankindâ€ variety. But for my part, Iâ€™m interested specifically in cloud computing, a field where change comes so quickly and where jobs are so specialized that itâ€™s near impossible for one person to stay on top of everything important. The rhetoric around â€œcloudâ€ might give you the impression that the term is practically synonymous with â€œinnovation,â€ but in talking to those in the trenches, the reality is a bit different.
I recently had a sit-down chat with Ping Li, a venture capitalist at Accel Partners who does investments across the layers of the cloud stack. Over the course of our conversation, Ping expressed frustration about the difficulty of hiring and maintaining talent right now. â€œItâ€™s this heated funding environment,â€ he said, going on to explain that all of the money sloshing around in the Valley had created a market for talent thatâ€™s just as tight as it was during the dotcom boom. Whatâ€™s worse, he explained, is that the talent shortage is stifling fundamental innovation in the cloud space.
To do really fundamental engineering innovation of the kind that was done, say, in the early days of Google and VMware, you need to hire and retain teams of talented engineers. But in todayâ€™s go-go funding environment, top engineers are being enticed with truckloads of money to break off and form two- and three-person startups. This phenomenon, explains Li, is why â€œmany of the really big innovations happen in less frothy times.â€ He did go on to clarify that â€œsome great companies do get created in these times (like Amazon in the last bubble). Itâ€™s just harder given talent shortage.â€
Liâ€™s comments are by no means the first Iâ€™ve heard in this vein. I recently talked to another well-known Valley entrepreneur who told me that his startup had poached top talent from a rival, only to see that talent leave and form a new VC-backed startup. Then thereâ€™s the email I got a few months ago from a friend of mine and product manager at Apple, who was wondering if I knew any cloud computing hackers that they could hire. When we get to the point where Apple product managers on the client side are reaching out to their personal networks in search of cloud coding talent for the worldâ€™s largest tech company, you know itâ€™s bad out there.
Iâ€™m facing this in my own project, a small book publishing startup where I moonlight as Chief Product Officer. Weâ€™re written in Ruby and hosted on Heroku, a pair of technical decisions that we made so that we could easily and painlessly scale, and so that we wouldnâ€™t have to waste resources on any sort of sysadmin work. Back in the depth of the last downturn, we were fortunate to have found a team of contract developers who are very talented and who are now therefore very, very busy.
This is par for the course. Weâ€™ve heard other startups complain that all of the big Ruby shops are taking only large jobs right now, because theyâ€™re so maxed out that thereâ€™s no bandwidth for small startup projects or overflow. So if our team gets run over by a bus on the way to an off-site, or vacuumed up by a VC with a giant bankroll, then all of the cloud-based redundancy and scalability in the world wonâ€™t get new features pushed out on our platform.
Talent as a hidden barrier to cloud adoption
The tight state of the current talent market has another, second-order effect on cloud innovation that goes beyond the team size issue that Ping Li points out. The talent shortage is also a factor in the slow pace of cloud adoption in the enterprise, or, at least, thatâ€™s one way of reading the results of Symantecâ€™s newly released State of the Cloud 2011 survey. The survey asked IT departments about their staffâ€™s readiness to make the leap into the cloud, and hereâ€™s what it found:
About half of the organizations surveyed said their IT staff is not ready for the move to cloud. While a handful (between 15 and 18 percent) rated their staff as extremely prepared, roughly half rated their IT staff as less than somewhat prepared. Part of the reason for this hesitancy is their staffâ€™s lack of experience. Less than 1 in 4 computer staffers have cloud experience. As discussed earlier, the adoption of cloud changes how IT works, so experience is absolutely crucial for IT.
The survey goes on to find that despite the excitement around cloud, most companiesâ€™ cloud initiatives are stalled in the discussion/trial phase.
Mooreâ€™s Law: too much of a good thing?
I said at the outset that there are two factors working against innovation in the cloud, but so far Iâ€™ve only talked about the first: easy money. The second factor is Mooreâ€™s Law. Advances in microprocessor technology are producing far more hardware than programmers can collectively or individually program.
On the enterprise side, the move to the cloud is driven partly by uncertain economic conditions and is based on an infrastructure layer that offers more performance per dollar with every upgrade cycle. So as economic fear continues to reign and commodity datacenter hardware gets cheaper in cost-per-MIPS, the pull of the cloud will get even stronger. And more dollars flowing into cloud hardware means that the demand will grow for cloud-savvy talent that can put the hardware to productive use.
Not only does the profusion of cheap computing power create an aggregate demand for more cloud programmers, but it also taxes individual programmers like never before. Coding, deploying, and maintaining highly parallel cloud apps is hard. Itâ€™s notoriously challenging to architect for parallelism at the design level, because itâ€™s hard for mere mortals to reason about parallelized tasks, especially if theyâ€™re non-deterministic (and they often are). Itâ€™s also hard to meaningfully test software on todayâ€™s large, massively parallel clouds; the logistics are a huge challenge.
Are taxes the answer?
Clayton Christensen, author of The Innovatorâ€™s Dilemma, touches on some of thsee issues in a must-read interview on GigaOM. Christensen fingers the â€œhot moneyâ€ that flows into sectors looking for a quick return as one of the factors slowing down innovation. He advocates eliminating capital gains taxes on investments of eight years or more, so that VCs will be incentivized to invest for the long term, thereby increasing the quality of innovation by shifting the focus away from things that can be done quickly.
Regardless of what you think of taxesâ€”more of them or less of themâ€”as an answer to our innovation dilemma, it seems clear that the current crisis in the cloud is the product of too many dollars and transistors chasing too few coders and sysadmins. It will take a while for the latter to catch up with the formerâ€¦ unless, of course, another major downturn strikes. It seems ironic that less money could equal more innovation, but it wouldnâ€™t be the first time that a wave of downsizing and tight money boosted productivity.
Posted by Janine E. Mooney, Editor