Lessons From Our Frail & Aging Infrastructure

Greg Schulz, Founder, Server and StorageIO | 7/23/2010 | 23 comments

Greg Schulz
As a society, we face growing problems repairing and maintaining the vital infrastructure we once took for granted.

This infrastructure ranges from the roads and bridges we drive every day to airports and railroads, communications networks, and electrical power grids -- the list goes on and on. In some cases we have seen tragic events resulting in the loss of life, such as the I-35W bridge collapse in Minnesota -- not far from where I live. Others represent mere inconveniences, such as local or regional power outages.

Most of these incidents involve aging, worn-out physical infrastructure desperately in need of repair or replacement. But infrastructure doesn't have to be old -- or even physical -- to cause problems when it fails.

The IT systems and applications all around us form a digital infrastructure that most enterprises take for granted -- until it's not there. A single glitch in the system can affect millions of people and companies, whether it's an airline computer that causes flight delays, lost email or text messages, or unplanned downtime for Websites and cloud applications. The problem isn't theoretical: Blackberry, Carbonite Inc. , Facebook (Nasdaq: FB), Twitter Inc. , Google (Nasdaq: GOOG), and eBay Inc. (Nasdaq: EBAY) -- all of these services, and many others, have suffered recent outages.

This isn't a new phenomenon. In fact, it's all part of a continuing problem that combines rapid growth, cost-cutting, competitive pressures, and short-term thinking. And it all adds up to more "fail whales" than any of us would like to see.

These applications are often designed, built, deployed, and managed with a throwaway mentality: "We'll replace it soon or throw it away and start over -- and if we don't we can leave the mess for somebody else to clean up."

Take a step back to the dotcom era a decade ago, when you heard the same "build it now, fix it later" mantra. Guess what? For the companies that survived, "fix it later" means fixing it now. These are today's legacy systems, but the people who designed them are long gone to work on the next cutting-edge project.

It's deja vu all over again, folks. Today's IT disruptions are tied to the same old application development, deployment, and scalability challenges we've been dealing with for decades. The question now is whether we're going to repeat the same mistakes again and again, without learning anything from them.

I think that's a real possibility. We live in an era of throwaway electronics and shiny new toys (SNTs). We throw away our still-new PCs for slightly newer ones. Then we throw those away for iPads. A new iPhone means we throw away the last one, and a new platform like Android means we throw that one away, too. It's a mindset that can easily extend to the underlying infrastructure, which can suffer from not being invested in or kept up to date.

There is some good news here. A few digital infrastructure carriers such as AT&T Inc. (NYSE: T) are investing billions to upgrade and optimize their networks and services. [Disclosure: I have used AT&T for about 15 years.] My local carrier, Qwest Communications International Inc. (NYSE: Q), has also been upgrading its networks -- granted, not as fast or as much as I would like, but they're still moving in the right direction.

Enterprises have to be aware that the existing public and private infrastructure you rely on may be more vulnerable than you expect. That also means now may be the time for infrastructure updates and improvements to things that many not qualify as "SNTs" but which we still rely on every day.

Just as important, enterprises must incorporate the lesson of infrastructure frailty into their own internal infrastructure choices. It's trendy -- and tempting -- to simply replace the old with the new, but it's critical to make sure that the replacement technology you choose will be cost-effective to operate and maintain over the long haul.

How do you do that? Leverage the hard lessons we've learned from designing, building, deploying, and managing resilient infrastructure, whether it's transportation systems, communications networks, bridges and roads, or information technology. Focus on single points of failure, and leverage Infrastructure Resource Management (IRM) processes and best practices.

This may not sound particularly appetizing in an age of severe competition and cost pressures. But there really isn't much choice. You can either pay up front to get this right, or you can pay even more later with downtime and service disruptions added to the bill. Either way, there will be a price to pay.

View Comments: Newest First | Oldest First | Threaded View
Page 1 / 3   >   >>
fowler   Lessons From Our Frail & Aging Infrastructure   7/27/2010 5:34:51 PM
job fair
How many young people will prioritize job security over adventure. The mainframe business is many things these days, but sexy it is not. Young talent is more likely to consider startups than insurance. In fifteen years they will regret it unless they make a mint selling their company to Google.

Somehow I don't see an insurance company's recruitment package standing up to what Google offers. Again, I am talking top engineering talent.
Matthew McKenzie   Lessons From Our Frail & Aging Infrastructure   7/27/2010 10:24:36 AM
Re: Building for the future, but the customer decide
@Broadway: That would have to be some younger person's dream job: Lots of job security (you can replace the person but not the skills) and a salary that would have to be way above the industry average.
DBK   Lessons From Our Frail & Aging Infrastructure   7/26/2010 11:30:23 PM
The customer decides or has the decision made for them?

There are so many business decisions that need to be made based on ROI and TCO.  Some of those are hard dollars and some are soft dollars.  The intangibles can be hard to calculate, especially if you are afraid to venture down that road because of what you might reveal.  There are costs for upgrading and you can see those dollars.  But how do you calculate lost productivity, down time, maintaining capability with current software and hardware.  If you ask me there are greater risk for not keeping your architecture current than there are for cost avoidance.  I equate it to owning a car, something most people can relate to.  You don’t just buy a car and drive it without doing any upkeep.  You have to maintain it and then there is also a point where it makes more sense to replace it than it does to keep on driving it.  It is also better to make decisions in a controlled environment instead of responding to disaster recovery.

Broadway   Lessons From Our Frail & Aging Infrastructure   7/26/2010 10:24:03 PM
Re: Building for the future, but the customer decide
@Matt, the plan B at that point is to hope that your venerable IT vets are generous and gracious enough to share their knowledge without jealousy or fear of losing their jobs, and that you've recruited enough young talent who have then enjoyed the grand tutelege of those old foggie mainframe experts. These youngsters can then either wean the insurance enterprise off the mainframes eventually or, more likely, continue to figure out ways to gerry-rig new technologies and systems onto the legacy systems.

The insurance industry, however, like most mature industries is running into problems with the recruitment side of that equation. Besides the wince that the mere word "insurance" evokes in most college age kids, there simply is a numbers issue...there aren't enough Gen Xers and Millennials to replace the Baby Boomers.
Matthew McKenzie   Lessons From Our Frail & Aging Infrastructure   7/26/2010 2:33:09 PM
Re: Building for the future, but the customer decide
@zeppy: I have heard the Y2K problem described more than once as the IT industry's version of The Boy Who Cried 'Wolf!'. It would be interesting to know whether the fallout from that is still having an impact, and if so, how deep it runs.
Matthew McKenzie   Lessons From Our Frail & Aging Infrastructure   7/26/2010 2:31:15 PM
Re: Building for the future, but the customer decide
...those with the wisdom about the legacy systems are not permitted to retire, or if they do, they are forced to sign back on as independent contractors.

@broadway: That actually sounds like a scary situation to me. Eventually, those older guys won't or can't provide their expertise, if nothing else they'll die of old age. What's the Plan B at that point. It'll be too late at that point to get rid of the legacy systems, and it might be too late to transfer the relevant knowledge to younger people.

Zentropist   Lessons From Our Frail & Aging Infrastructure   7/26/2010 12:13:37 PM
Re: Lessons we should have already learned
@Zeppy: both you and CMTucker are absolutely right in your comments regarding our physical infrastructure and its vulnerabilities. So much of the physical trappings that made us a modern, First World nation were last build during the Eisenhower Administration, and as our population continues to swell, and we've exponentially gained more vehicles, devices requiring electric power, etc., we've put tremendous demands on this infrastructure to keep up with growth.

I'm as much against the rampant and irresponsible government spending as anyone else with a modicum of fiscal sense, but our failure to ensure that we can meet the demands of our population leads me to believe that we will probably increasingly see failures of it in upcoming years.

If there's any truth to the notion of high levels of solar flare activity starting next year, we'll see how out telecom and electrical grid handles that challenge...

zeppy   Lessons From Our Frail & Aging Infrastructure   7/26/2010 11:37:36 AM
Re: Lessons we should have already learned
@CMTucker, I have to agree and I gather you're not just talking about IT infrastructue here. I was shocked and amazed to read recently of how common it is becoming across the U.S. to abandon previously paved roads in rural areas. As municipalities are dealing with tighter and tighter budgets, rather than maintaining the paved roads in lesser used areas, it is becoming more common to revert to packed gravel roads. I think that's a pretty striking example that our society is crumbling in some very serious ways.
zeppy   Lessons From Our Frail & Aging Infrastructure   7/26/2010 11:27:54 AM
Re: Building for the future, but the customer decide
This is a really interesting discussion, thanks for all the insightful comments! @fowler, I think your point about the Y2K bug is a telling one to this whole infrastructure problem. For a brief moment in time (or so it seemed anyway) the industry (and the world) thought the IT infrastructure was in serious trouble. Managers and executives got involved, some frantic coding and patching was done, but the overall sense was this problem with our aging infrastructure was going to cause untold problems and huge costs. And then the time came, and nothing catastrophic happened. So all those managers and executives who for once had to think and worry about infrastucture, went back to not worrying about it, and here we are. (Simplistic, I know, but I'm not really sure how to explain the general lack of interest in maintaining IT infrastructure in most C-suites).

As an aside, I was on a long-term assignment in New Zealand when Y2K hit, and while it was clear that most of the U.S. and the developed world was worried and stressing about this particular moment and what it could mean, the kiwis looked it as a chance to throw a massive party to be seem by the whole world, since they were in the first populated area where the clock turned over to 2000. It was one hell of a party.

CMTucker   Lessons From Our Frail & Aging Infrastructure   7/25/2010 7:50:31 PM
Lessons we should have already learned
This is a big problem. Our infrastructure is crumbling around us. Yet, when "someone" talks about spending some money on it..."someone else" starts whining about the debt...while at the same time telling rich folks they don't have to pay their taxes.

It'll be a third world feudal society if we don't wake up and start to tell the difference between fact and fiction.
Page 1 / 3   >   >>

The blogs and comments posted on EnterpriseEfficiency.com do not reflect the views of TechWeb, EnterpriseEfficiency.com, or its sponsors. EnterpriseEfficiency.com, TechWeb, and its sponsors do not assume responsibility for any comments, claims, or opinions made by authors and bloggers. They are no substitute for your own research and should not be relied upon for trading or any other purpose.

More Blogs from Greg Schulz
Greg Schulz   4/23/2010   49 comments
Last night, the 75th annual National Football League (NFL) draft officially kicked off. All of the buzz and media coverage -- the prospects, who will get drafted when, whether or not they ...
Latest Archived Broadcast
In this episode, you'll learn how to stretch the limits of your private cloud -- and how to recognize the limits that can't be exceeded.
On-demand Video with Chat
IT has to deploy Server 2012 in a way that fits the architecture of its application delivery system.
E2 IT Migration Zones
IT Migration Zone - UK
Why PowerShell Is Important
Reduce the Windows 8 Footprint for VDI
Rethinking Storage Management
IT Migration Zone - FR
SQL Server : 240 To de mémoire flash pour votre data warehouse
Quand Office vient booster les revenus Cloud et Android de Microsoft
Windows Phone : Nokia veut davantage d'applications (et les utilisateurs aussi)
IT Migration Zone - DE
Cloud Computing: Warum Unternehmen trotz NSA auf die „private“ Wolke setzen sollten
Cloud Computing bleibt Wachstumsmarkt – Windows Azure ist Vorreiter
Like Us on Facebook
Twitter Feed
Enterprise Efficiency Twitter Feed
Site Moderators Wanted
Enterprise Efficiency is looking for engaged readers to moderate the message boards on this site. Engage in high-IQ conversations with IT industry leaders; earn kudos and perks. Interested? E-mail:
[email protected]
Informed CIO: Dollars & Sense: Virtual Desktop Infrastructure
Cut through the VDI hype and get the full picture -- including ROI and the impact on your Data Center -- to make an informed decision about your virtual desktop infrastructure deployments.

Read the full report
Virtualization Management: Time To Get Serious
Welcome to the backside of the virtualization wave. Discover the state of virtualization management and where analysts are predicting it is heading

Read the full report
A Video Case Study – Translational Genomics Research Institute
e2 Storage Video

On the Case
TGen IT: Where We're Going Next

7|11|12   |   08:12   |   10 comments

Now that TGen has broken new ground in genomic research by using Dell's storage, cloud, and high-performance computing solutions, the company discusses what will come next for it and for personalized medicine.
On the Case
Better Care Through Better Communications

6|6|12   |   02:24   |   11 comments

The achievements of the TGen/Dell project could improve how all people receive healthcare, because they are creating ways to improve end-to-end communication of medical data.
On the Case
TGen IT: Where We Are Now

5|15|12   |   06:58   |   6 comments

TGen is breaking new ground in genomic research by using Dell's storage, cloud, and high-performance computing solutions.
On the Case
TGen IT: Where We Were

4|27|12   |   06:45   |   10 comments

The Translational Genomics Research Institute wanted to save lives, but its efforts were hobbled by immense computing challenges related to collecting, processing, sharing, and storing enormous amounts of data.
On the Case
1,200% Faster

4|18|12   |   02:27   |   12 comments

Through their partnership, Dell and TGen have increased the speed of TGen’s medical research by 1,200 percent.
On the Case
IT May Improve Children's Chances of Survival

4|17|12   |   02:12   |   8 comments

IT is helping medical researchers reach breakthroughs in a way and pace never seen before.
On the Case
Medical Advances in the Cloud

4|10|12   |   1:25   |   5 comments

TGen and Dell are pushing the boundaries of computing, and harnessing the power of the cloud to improve healthcare.
On the Case
TGen: Living the Mission

4|9|12   |   2:25   |   3 comments

TGen's CIO puts the organizational mission at the heart of everything the IT staff does.
On the Case
TGen Speeding Up Biomedical Research to Save More Lives

4|5|12   |   1:59   |   6 comments

The Translational Genomics Research Institute is revamping its computing to improve speed, storage, and collaboration – and, most importantly, to save lives.
On the Case
Computing Power Helping to Save Children's Lives

3|28|12   |   2:13   |   3 comments

The Translational Genomics Institute’s partnership with Dell is enabling them to treat kids with neuroblastoma more quickly and save more lives.
Tom Nolle
How Deep Is My Storage Hierarchy?

7|3|12   |   2:13   |   5 comments

At the GigaOM Structure conference, a startup announced a cloud and virtualization storage optimizing approach that shows there's still a lot of thinking to be done on the way storage joins the virtual world.
E2 Interview
What Other Industries Can Learn From Financial Services

6|13|12   |   02:08   |   3 comments

We asked CIO Steve Rubinow what CIOs in other industries can learn from the financial services industry about datacenter efficiency, security, and green computing.
E2 Interview
Removing Big-Data Flow Bottlenecks

6|12|12   |   02:55   |   No comments

We ask CIO Steve Rubinow what pieces of financial services infrastructure need to perform better to get traders info faster.
E2 Interview
Getting Traders the Data They Need

6|11|12   |   02:04   |   1 comment

We ask CIO Steve Rubinow: What do stock market traders need to know, how fast do they need it, and how can CIOs get it to them?
E2 Interview
Can IT Help Fix the Global Economy?

6|8|12   |   02:32   |   2 comments

We ask CIO Steve Rubinow whether today's IT can help repair the global economy (and if IT played any role in the economy's collapse).
E2 Interview
More Competitive Business via Datacenter Strategy

5|4|12   |   2:46   |   1 comment

Businesses need to be competitive, yet efficient, and both goals affect datacenter design.
E2 Interview
The Recipe for Greater Efficiency

5|3|12   |   3:14   |   2 comments

Intel supplies the best ingredients to drive greater datacenter efficiency and support new compute, storage, and networking needs.
E2 Interview
Datacenters Enabling Business Transformation

5|1|12   |   06:37   |   1 comment

Dell’s Gaurav Chand says that for the first time ever datacenter technology is truly enabling all kinds of organizations to transform their business and achieve new objectives.
Tom Nolle
Cloud Data: Big AND Persistent!

3|28|12   |   2:11   |   10 comments

We always hear about "Big" data, but a real issue in cloud storage is not just bigness but also persistence. A large data model is less complicated than a big application repository that somehow needs to be accessed. The Hadoop send-program-to-data model may be the answer.
Tom Nolle
Project Lightning Streamlines Storage

2|16|12   |   2:09   |   2 comments

EMC's Project Lightning has matured into a product set, and it's important, less because it has new features or capabilities in storage technology and management, than because it may package the state of the art in a way more businesses can deploy.
Tom Nolle
Big Data Appliance Is Big News

1|12|12   |   2:18   |   No comments

Oracle's release of a Hadoop appliance for Big Data may be a signal that we're shifting to database appliances.
Tom Nolle
Myopia Can Hurt Storage Policy

12|22|11   |   2:08   |   No comments

We're at the beginning of a cloud-driven revolution in storage, but Oracle's quarter shows that enterprises are hunkering down on old concepts because they're afraid of the costs in the near term.
Sara Peters
An Untrained User & a Mobile Medical Device

12|19|11   |   2:43   |   11 comments

Untrained end users, clueless central IT staff, and expensive mobile devices are a worrisome combination for healthcare CIOs.
Tom Nolle
Too Many Labels on 'Big Data'?

12|9|11   |   2:12   |   3 comments

However you label it, structured and unstructured information are different and will likely always require different tools.
Sara Peters
E2 Debuts New Storage Section

12|8|11   |   1:51   |   1 comment

Need strategic guidance on everything from SSDs to 100 percent virtualized datacenters? Look no further.