Lessons From Our Frail & Aging Infrastructure

Greg Schulz, Founder, Server and StorageIO | 7/23/2010 | 23 comments
Digg   Del.icio.us   Reddit   Email This   Tweet This

As a society, we face growing problems repairing and maintaining the vital infrastructure we once took for granted.

This infrastructure ranges from the roads and bridges we drive every day to airports and railroads, communications networks, and electrical power grids -- the list goes on and on. In some cases we have seen tragic events resulting in the loss of life, such as the I-35W bridge collapse in Minnesota -- not far from where I live. Others represent mere inconveniences, such as local or regional power outages.

Most of these incidents involve aging, worn-out physical infrastructure desperately in need of repair or replacement. But infrastructure doesn't have to be old -- or even physical -- to cause problems when it fails.

The IT systems and applications all around us form a digital infrastructure that most enterprises take for granted -- until it's not there. A single glitch in the system can affect millions of people and companies, whether it's an airline computer that causes flight delays, lost email or text messages, or unplanned downtime for Websites and cloud applications. The problem isn't theoretical: Blackberry, Carbonite Inc. , Facebook , Twitter Inc. , Google (Nasdaq: GOOG), and eBay Inc. (Nasdaq: EBAY) -- all of these services, and many others, have suffered recent outages.

This isn't a new phenomenon. In fact, it's all part of a continuing problem that combines rapid growth, cost-cutting, competitive pressures, and short-term thinking. And it all adds up to more "fail whales" than any of us would like to see.

These applications are often designed, built, deployed, and managed with a throwaway mentality: "We'll replace it soon or throw it away and start over -- and if we don't we can leave the mess for somebody else to clean up."

Take a step back to the dotcom era a decade ago, when you heard the same "build it now, fix it later" mantra. Guess what? For the companies that survived, "fix it later" means fixing it now. These are today's legacy systems, but the people who designed them are long gone to work on the next cutting-edge project.

It's deja vu all over again, folks. Today's IT disruptions are tied to the same old application development, deployment, and scalability challenges we've been dealing with for decades. The question now is whether we're going to repeat the same mistakes again and again, without learning anything from them.

I think that's a real possibility. We live in an era of throwaway electronics and shiny new toys (SNTs). We throw away our still-new PCs for slightly newer ones. Then we throw those away for iPads. A new iPhone means we throw away the last one, and a new platform like Android means we throw that one away, too. It's a mindset that can easily extend to the underlying infrastructure, which can suffer from not being invested in or kept up to date.

There is some good news here. A few digital infrastructure carriers such as AT&T Inc. (NYSE: T) are investing billions to upgrade and optimize their networks and services. [Disclosure: I have used AT&T for about 15 years.] My local carrier, Qwest Communications International Inc. (NYSE: Q), has also been upgrading its networks -- granted, not as fast or as much as I would like, but they're still moving in the right direction.

Enterprises have to be aware that the existing public and private infrastructure you rely on may be more vulnerable than you expect. That also means now may be the time for infrastructure updates and improvements to things that many not qualify as "SNTs" but which we still rely on every day.

Just as important, enterprises must incorporate the lesson of infrastructure frailty into their own internal infrastructure choices. It's trendy -- and tempting -- to simply replace the old with the new, but it's critical to make sure that the replacement technology you choose will be cost-effective to operate and maintain over the long haul.

How do you do that? Leverage the hard lessons we've learned from designing, building, deploying, and managing resilient infrastructure, whether it's transportation systems, communications networks, bridges and roads, or information technology. Focus on single points of failure, and leverage Infrastructure Resource Management (IRM) processes and best practices.

This may not sound particularly appetizing in an age of severe competition and cost pressures. But there really isn't much choice. You can either pay up front to get this right, or you can pay even more later with downtime and service disruptions added to the bill. Either way, there will be a price to pay.

Digg   Del.icio.us   Reddit   Email This   Tweet This
Post a Comment
View comments: newest first | oldest first | threaded
Page 1 of 3   Next >   Last >>
fowler   Lessons From Our Frail & Aging Infrastructure   7/27/2010 5:34:51 PM
job fair
How many young people will prioritize job security over adventure. The mainframe business is many things these days, but sexy it is not. Young talent is more likely to consider startups than insurance. In fifteen years they will regret it unless they make a mint selling their company to Google.

Somehow I don't see an insurance company's recruitment package standing up to what Google offers. Again, I am talking top engineering talent.
Matthew McKenzie   Lessons From Our Frail & Aging Infrastructure   7/27/2010 10:24:36 AM
Re: Building for the future, but the customer decide
@Broadway: That would have to be some younger person's dream job: Lots of job security (you can replace the person but not the skills) and a salary that would have to be way above the industry average.
DBK   Lessons From Our Frail & Aging Infrastructure   7/26/2010 11:30:23 PM
The customer decides or has the decision made for them?

There are so many business decisions that need to be made based on ROI and TCO.  Some of those are hard dollars and some are soft dollars.  The intangibles can be hard to calculate, especially if you are afraid to venture down that road because of what you might reveal.  There are costs for upgrading and you can see those dollars.  But how do you calculate lost productivity, down time, maintaining capability with current software and hardware.  If you ask me there are greater risk for not keeping your architecture current than there are for cost avoidance.  I equate it to owning a car, something most people can relate to.  You don’t just buy a car and drive it without doing any upkeep.  You have to maintain it and then there is also a point where it makes more sense to replace it than it does to keep on driving it.  It is also better to make decisions in a controlled environment instead of responding to disaster recovery.

Broadway   Lessons From Our Frail & Aging Infrastructure   7/26/2010 10:24:03 PM
Re: Building for the future, but the customer decide
@Matt, the plan B at that point is to hope that your venerable IT vets are generous and gracious enough to share their knowledge without jealousy or fear of losing their jobs, and that you've recruited enough young talent who have then enjoyed the grand tutelege of those old foggie mainframe experts. These youngsters can then either wean the insurance enterprise off the mainframes eventually or, more likely, continue to figure out ways to gerry-rig new technologies and systems onto the legacy systems.

The insurance industry, however, like most mature industries is running into problems with the recruitment side of that equation. Besides the wince that the mere word "insurance" evokes in most college age kids, there simply is a numbers issue...there aren't enough Gen Xers and Millennials to replace the Baby Boomers.
Matthew McKenzie   Lessons From Our Frail & Aging Infrastructure   7/26/2010 2:33:09 PM
Re: Building for the future, but the customer decide
@zeppy: I have heard the Y2K problem described more than once as the IT industry's version of The Boy Who Cried 'Wolf!'. It would be interesting to know whether the fallout from that is still having an impact, and if so, how deep it runs.
Matthew McKenzie   Lessons From Our Frail & Aging Infrastructure   7/26/2010 2:31:15 PM
Re: Building for the future, but the customer decide
...those with the wisdom about the legacy systems are not permitted to retire, or if they do, they are forced to sign back on as independent contractors.

@broadway: That actually sounds like a scary situation to me. Eventually, those older guys won't or can't provide their expertise, if nothing else they'll die of old age. What's the Plan B at that point. It'll be too late at that point to get rid of the legacy systems, and it might be too late to transfer the relevant knowledge to younger people.

Zentropist   Lessons From Our Frail & Aging Infrastructure   7/26/2010 12:13:37 PM
Re: Lessons we should have already learned
@Zeppy: both you and CMTucker are absolutely right in your comments regarding our physical infrastructure and its vulnerabilities. So much of the physical trappings that made us a modern, First World nation were last build during the Eisenhower Administration, and as our population continues to swell, and we've exponentially gained more vehicles, devices requiring electric power, etc., we've put tremendous demands on this infrastructure to keep up with growth.

I'm as much against the rampant and irresponsible government spending as anyone else with a modicum of fiscal sense, but our failure to ensure that we can meet the demands of our population leads me to believe that we will probably increasingly see failures of it in upcoming years.

If there's any truth to the notion of high levels of solar flare activity starting next year, we'll see how out telecom and electrical grid handles that challenge...

Z.
zeppy   Lessons From Our Frail & Aging Infrastructure   7/26/2010 11:37:36 AM
Re: Lessons we should have already learned
@CMTucker, I have to agree and I gather you're not just talking about IT infrastructue here. I was shocked and amazed to read recently of how common it is becoming across the U.S. to abandon previously paved roads in rural areas. As municipalities are dealing with tighter and tighter budgets, rather than maintaining the paved roads in lesser used areas, it is becoming more common to revert to packed gravel roads. I think that's a pretty striking example that our society is crumbling in some very serious ways.
zeppy   Lessons From Our Frail & Aging Infrastructure   7/26/2010 11:27:54 AM
Re: Building for the future, but the customer decide
This is a really interesting discussion, thanks for all the insightful comments! @fowler, I think your point about the Y2K bug is a telling one to this whole infrastructure problem. For a brief moment in time (or so it seemed anyway) the industry (and the world) thought the IT infrastructure was in serious trouble. Managers and executives got involved, some frantic coding and patching was done, but the overall sense was this problem with our aging infrastructure was going to cause untold problems and huge costs. And then the time came, and nothing catastrophic happened. So all those managers and executives who for once had to think and worry about infrastucture, went back to not worrying about it, and here we are. (Simplistic, I know, but I'm not really sure how to explain the general lack of interest in maintaining IT infrastructure in most C-suites).

As an aside, I was on a long-term assignment in New Zealand when Y2K hit, and while it was clear that most of the U.S. and the developed world was worried and stressing about this particular moment and what it could mean, the kiwis looked it as a chance to throw a massive party to be seem by the whole world, since they were in the first populated area where the clock turned over to 2000. It was one hell of a party.

 
CMTucker   Lessons From Our Frail & Aging Infrastructure   7/25/2010 7:50:31 PM
Lessons we should have already learned
This is a big problem. Our infrastructure is crumbling around us. Yet, when "someone" talks about spending some money on it..."someone else" starts whining about the debt...while at the same time telling rich folks they don't have to pay their taxes.

It'll be a third world feudal society if we don't wake up and start to tell the difference between fact and fiction.
Page 1 of 3   Next >   Last >>


More Blogs from Greg Schulz
Greg Schulz   4/23/2010   49 comments
Last night, the 75th annual National Football League (NFL) draft officially kicked off. All of the buzz and media coverage -- the prospects, who will get drafted when, whether or not they ...
Latest Archived Broadcast
Cliff Bell, veteran CIO and popular EnterpriseEfficiency.com blogger, shares his firsthand insights into the issues and concerns that give today's CIOs and IT leaders nightmares. From technology upgrades and migrations to cloud computing and security, Bell and E2 editor-in-chief Fredric Paul will discuss the most pressing and most promising solutions for enterprise IT.
Whitepapers and Analytics Reports
Defining Data Center Efficiency: Standardization Is Job One
Data center efficiency begins with standardization, which can lower TCO, cut energy consumption, and help control acquisition costs, among other benefits. Learn how to plan, implement, and manage a successful data center standardization strategy.

Read the full report
Defining Data Center Efficiency: Simplifying Through Virtualization
By making infrastructure and data centers more efficient, IT departments free up money for innovation. Virtualization is a vital tool for implementing a data center efficiency strategy, as illustrated by the tips in this comprehensive report.

Read the full report
Defining Data Center Efficiency: Taking Virtualization to the Next Level with Automation
Virtualization without automation leads to delays and missed opportunities. Find out how companies are automating on standardized, open x86 platforms to prepare for growth -- without ripping out and replacing existing data center infrastructure.

Read the full report
drill into Virtualization
BLOGS
What Happens When Virtualization Gets Real?
9/1/2010 - New products, new pronouncements, and new deals mark VMware's virtualization conference in San Francisco this week
Coursework
SPONSORED BY DELL
BRIEFINGS
CASE STUDIES
WHITE PAPERS
Site Moderators
Enterprise Efficiency is looking for engaged readers to moderate the message boards on this site. Engage in high-IQ conversations with IT industry leaders; earn kudos and perks. Interested? E-mail:
moderators@enterpriseefficiency.com

Enabling People and Organizations to Harness the Transformative Power of Technology