What I Was Thinking... When Budget Cuts Made the Backup Fail

Michael Hugos, Principal, Center for Systems Innovation | 7/25/2013 | 12 comments

Michael Hugos
Editor's Note: This is part of a series written by CIOs discussing their thought processes and lessons learned from major events in their tenures as CIO. Check back tomorrow for a companion post.

I was once in charge of IT operations at a company where the decision had been made to defer as many capital expenses as possible. That meant all the VPs and directors went through our budgets and identified infrastructure upgrades or new systems that could be postponed for a year or two. I did my duty and found several projects that could be postponed. One of them was upgrading the emergency power generator that provided backup power for our datacenter and headquarters building. I soon wished i hadn't been asked to make that decision.

The summer that year turned out to be hotter than normal. One hot day in late August, we began experiencing power brownouts in the office park where our headquarters was located. Suddenly, right after lunch, the power went off completely. The building went dark and quiet.

I stepped out of my office and looked down the hall toward the datacenter. To my relief, the lights were on in the datacenter, and our systems stayed up. I should have heard the roar of the generator behind the building as it kicked in to supply the power we needed, yet the generator did not come on. We quickly realized our systems were running on the batteries of the UPS unit. We had about 30 minutes of power. If the commercial power grid didn't come back on in the next five minutes or so, we would have to start an orderly shutdown of all our systems while we still had time.

I went down the hall to the datacenter and got ready to give the order to shut down our systems. That meant not only all the company's internal systems, such as ERP and HR, but also all our Internet systems, such as our website, our e-commerce system, and our EDI and other data links. If these went down, you know what would hit the fan in a big way, and I would be the one responsible. There were career-limiting consequences involved. No one had much to say; we stood around thinking about what we had to do. I looked at my watch. There really is nothing to say at times like this.

After about three and a half long minutes, the power did come back on. We looked at each other and thanked our lucky stars. My knees were weak. Then we rushed out back to check on the generator. On the first Wednesday of every month, we had a regularly scheduled operation where the generator was switched on and run for 30 minutes to make sure it was in working order. We couldn't understand why it hadn't worked when we really needed it.

Right away, we called a service technician, who came out to check. He found that, although the generator turned on when we ran our monthly tests, the failover switch that sensed when the commercial power grid went down was not working. We hadn't been testing that.

We replaced the switch immediately and ran another test. We turned off power to the building to see if the failover switch would work and trigger the generator to start up. The generator did start up, but within minutes, it overheated and shut itself down. Then we remembered that the generator had been installed years ago to support a smaller operation. Since then, more people had been hired, and more gear from servers to PCs and air conditioners had been added. Now when the generator was subjected to that increased load, it choked.

This started a sequence of tuneups, repairs, and tinkering to try to get another year or two out of the old generator. We looked at having it power only the datacenter, but the rest of the headquarters would be dark, so nobody could work anyway. None of the other managers liked that idea. After a couple of days and several more tests, I realized the old generator just could not be trusted. And I felt like I was tempting fate by delaying the decision to buy a new generator.

I drew up a capital expense request for a new generator -- not just a replacement for what was already there, but a larger generator and some related upgrades to the UPS unit and the electrical wiring in the datacenter. I was about to learn a lesson in the logic and the politics of requesting a large, unbudgeted capital expense during a tough year when profits are down and money is tight. Come back tomorrow to see what I learned.

Related posts:

View Comments: Newest First | Oldest First | Threaded View
Page 1 / 2   >   >>
batye   What I Was Thinking... When Budget Cuts Made the Backup Fail   3/1/2014 5:19:36 PM
Re: Power shutdown
could not agree more interesting point, as proper testing is a must... and a lot of the factors need to be taken in the account...
eethtworkz   What I Was Thinking... When Budget Cuts Made the Backup Fail   7/31/2013 2:32:19 PM
Re: Power shutdown
Umair,

Quaint notion for sure.

If the tests were carried out with Good coordination-What level of protection could we have achieved here?

My bet is more than 90percent.

It is definitely better than the End result here.

Will Enterprise IT heed it?

I certainly hope so.

 
Michael Hugos   What I Was Thinking... When Budget Cuts Made the Backup Fail   7/29/2013 5:20:33 PM
Re: Test vs Actual
Hi Umair,

Yes you describe the problem well - reality often does not play out the way you expected it to. You drill for what you expect to happen and then something else happens. 

So in addition to specific drills, I'm also big on making sure my IT group has good problem solving skills. When problems occur that we had not expected, it is our ability to think on our feet in the moment and not panic that saves the day.
User Ranking: Blogger
Umair Ahmed   What I Was Thinking... When Budget Cuts Made the Backup Fail   7/29/2013 5:14:02 PM
Re: Power shutdown
@ eethworkz: Great observation, I agree with you.  The failure seems to be impact of the test of insufficient intensity and time length. If the test were carried out with good coordination, the weakness could have been identified before the real failure. 
Umair Ahmed   What I Was Thinking... When Budget Cuts Made the Backup Fail   7/29/2013 4:53:55 PM
Test vs Actual
Very interesting post, Michael. Glad that your luck worked and you got the power back in time. The story very well explains the difference between artificial and real scenarios. Even the very well planned tests and drills miss some real aspects of the disaster and so fail to reveal some very important weaknesses. 
SunitaT   What I Was Thinking... When Budget Cuts Made the Backup Fail   7/26/2013 7:45:19 AM
Re: Power shutdown
Here in our country where power shutdown is very common, one would have never delayed the decision of buying a new generator. 

@Hammad, I think its always better to plan for the worstcase and keep the backup ready so that in any eventuality project execution is not impacted. No doubt extra backup will require extra investment but project execution wont get affected.
Don K   What I Was Thinking... When Budget Cuts Made the Backup Fail   7/26/2013 6:43:34 AM
Re: Power shutdown
@Hayder: True but also the quality of it too should be considered since things cannot be rushed just because its urgent. It should match with quality as well.
eethtworkz   What I Was Thinking... When Budget Cuts Made the Backup Fail   7/26/2013 3:55:45 AM
Re: Power shutdown
Zaius,

I don't think its fair to blame the Power Management Guys entirely for this Issue.

Lets face the facts;there was also a breakdown of Coordination here between various members of the entire IT Team.

If the Power People knew in advance about Load Spikes I am sure they would have taken all the neccessary precautions.

Regards

Ashish.

 
Zaius   What I Was Thinking... When Budget Cuts Made the Backup Fail   7/26/2013 12:27:45 AM
Re: Power shutdown
This was some lucky event for you. It did not actually cause the complete balckout but yet it shook the entire settlement. And, made everyone concerned about the risks and possible outcomes.

However, I wonder, the person responsible for power management did not do a good job. When a building is consuming more loads (more servers, more people, more air conditioners), the concerned person should raise the flag that the generator is dwarfed by the new demands.
vnewman   What I Was Thinking... When Budget Cuts Made the Backup Fail   7/25/2013 3:30:06 PM
Re: Power shutdown
Wow!  What a scary story - my stomach turned just reading about it and putting myself in your shoes.  Just like with airplane crashes, it is always the convergence of a series of mishaps or failures that can send the system into a tailspin.

To be honest though - when a similar, albiet not as dire situation occured at my firm due to budget cuts, the first thing I thought was, "Seeeeeee...I told you so!!!!"
Page 1 / 2   >   >>


The blogs and comments posted on EnterpriseEfficiency.com do not reflect the views of TechWeb, EnterpriseEfficiency.com, or its sponsors. EnterpriseEfficiency.com, TechWeb, and its sponsors do not assume responsibility for any comments, claims, or opinions made by authors and bloggers. They are no substitute for your own research and should not be relied upon for trading or any other purpose.

More Blogs from Michael Hugos
Michael Hugos   3/11/2014   9 comments
CIOs and COOs generally play a more supporting role than their more glamorous counterparts in marketing, sales, and finance. You can turn that to an advantage if you want to build a better ...
Michael Hugos   3/5/2014   15 comments
CFOs can appear to be a smug and self-assured bunch, but underneath that smug exterior there lurks much uncertainty and doubt.
Michael Hugos   2/25/2014   26 comments
In our continuing series about whom the CIO should make friends with, the next most pressured person in your company (behind the CMO) is probably the VP of Sales. There might be ways to ...
Michael Hugos   2/19/2014   21 comments
If you are a CIO who wants to ensure your place in the organization, a good place to start is with the CMO. That is because the CMO is most likely the C-suite executive under the most ...
Michael Hugos   2/5/2014   30 comments
At present we are living in the twilight of the Industrial Age from the last century, just as people in 1914 were living in the twilight of the Victorian Age from the century before that. ...
Latest Archived Broadcast
We talk with Bernard Golden about accelerating application delivery in the cloud.
On-demand Video with Chat
Register for this video discussion to learn how tablets can provide true business usability and productivity.
E2 IT Migration Zones
IT Migration Zone - UK
Why PowerShell Is Important
Reduce the Windows 8 Footprint for VDI
Rethinking Storage Management
IT Migration Zone - FR
SQL Server : 240 To de mémoire flash pour votre data warehouse
Quand Office vient booster les revenus Cloud et Android de Microsoft
Windows Phone : Nokia veut davantage d'applications (et les utilisateurs aussi)
IT Migration Zone - DE
Cloud Computing: Warum Unternehmen trotz NSA auf die „private“ Wolke setzen sollten
Cloud Computing bleibt Wachstumsmarkt – Windows Azure ist Vorreiter
Like Us on Facebook
Twitter Feed
Enterprise Efficiency Twitter Feed
Site Moderators Wanted
Enterprise Efficiency is looking for engaged readers to moderate the message boards on this site. Engage in high-IQ conversations with IT industry leaders; earn kudos and perks. Interested? E-mail:
[email protected]
Dell's Efficiency Modeling Tool
The major problem facing the CIO is how to measure the effectiveness of the IT department. Learn how Dell’s Efficiency Modeling Tool gives the CIO two clear, powerful numbers: Efficiency Quotient and Impact Quotient. These numbers can be transforma¬tive not only to the department, but to the entire enterprise.

Read the full report
The State of Enterprise Efficiency in the Virtual Era: Virtualization – Smart Approaches to Maximize Gains
Virtualization is a presence in nearly all enterprise data centers. But not all companies are using it to its best effect. Learn the common characteristics of success, what barriers companies face, and how to get the most from your efforts.

Read the full report
Informed CIO: Dollars & Sense: Virtual Desktop Infrastructure
Cut through the VDI hype and get the full picture -- including ROI and the impact on your Data Center -- to make an informed decision about your virtual desktop infrastructure deployments.

Read the full report
SPONSORED BY DELL
CASE STUDIES
EBOOKS
PUBLIC SECTOR RESOURCES
VIDEOS
WHITE PAPERS
WINDOWS SERVER 2012 RESOURCES
A Video Case Study – Translational Genomics Research Institute
e2 Video


On the Case
TGen IT: Where We're Going Next

7|11|12   |   08:12   |   10 comments


Now that TGen has broken new ground in genomic research by using Dell's storage, cloud, and high-performance computing solutions, the company discusses what will come next for it and for personalized medicine.
On the Case
Better Care Through Better Communications

6|6|12   |   02:24   |   11 comments


The achievements of the TGen/Dell project could improve how all people receive healthcare, because they are creating ways to improve end-to-end communication of medical data.
On the Case
TGen IT: Where We Are Now

5|15|12   |   06:58   |   6 comments


TGen is breaking new ground in genomic research by using Dell's storage, cloud, and high-performance computing solutions.
On the Case
TGen IT: Where We Were

4|27|12   |   06:45   |   10 comments


The Translational Genomics Research Institute wanted to save lives, but its efforts were hobbled by immense computing challenges related to collecting, processing, sharing, and storing enormous amounts of data.
On the Case
1,200% Faster

4|18|12   |   02:27   |   12 comments


Through their partnership, Dell and TGen have increased the speed of TGen’s medical research by 1,200 percent.
On the Case
IT May Improve Children's Chances of Survival

4|17|12   |   02:12   |   8 comments


IT is helping medical researchers reach breakthroughs in a way and pace never seen before.
On the Case
Medical Advances in the Cloud

4|10|12   |   1:25   |   5 comments


TGen and Dell are pushing the boundaries of computing, and harnessing the power of the cloud to improve healthcare.
On the Case
TGen: Living the Mission

4|9|12   |   2:25   |   3 comments


TGen's CIO puts the organizational mission at the heart of everything the IT staff does.
On the Case
TGen Speeding Up Biomedical Research to Save More Lives

4|5|12   |   1:59   |   6 comments


The Translational Genomics Research Institute is revamping its computing to improve speed, storage, and collaboration – and, most importantly, to save lives.
On the Case
Computing Power Helping to Save Children's Lives

3|28|12   |   2:13   |   3 comments


The Translational Genomics Institute’s partnership with Dell is enabling them to treat kids with neuroblastoma more quickly and save more lives.
Tom Nolle
The Big Reason to Use Office

3|18|14   |   02:24   |   46 comments


Office and personal productivity tools come in a first-class and coach flavor set, but what makes the difference is primarily little things that most users won't encounter. What's the big issue in using something other than Office, and can you get around it?
E2 Editors
SPONSORED: Mobile Security — A Use Case

3|4|14   |   04:27   |   16 comments


New mobile security solutions can accommodate a wide array of needs, including those of a complex university environment.
Tom Nolle
Killing Net Neutrality Might Save You Money

1|16|14   |   2:13   |   16 comments


The DC Court of Appeals voided most of the Neutrality Order, and whatever it might mean for the Internet overall, it might mean better and cheaper Internet VPNs for businesses.
Tom Nolle
The Internet of Everythinguseful

1|10|14   |   2:18   |   19 comments


We really don't want an "Internet of Everything" but even building an Internet of Everythinguseful means setting some ground rules to insure there's value in the process and that costs and risks are minimized.
Tom Nolle
Maturing Google Chrome

12|30|13   |   2.18   |   25 comments


Google's Chrome OS has a lot of potential value and a lot of recent press, but it still needs something to make it more than a thin client. It needs cloud integration, it needs extended APIs via web services, and it needs to suck it up and support a hard drive.
Sara Peters
No More Cookie-Cutter IT

12|23|13   |   03.58   |   21 comments


Creating the right combination of technology, people, and processes for your IT organization is a lot like baking Christmas cookies.
Sara Peters
Smart Wigs Not a Smart Idea

12|5|13   |   3:01   |   46 comments


Sony is seeking a patent for wigs that contain computing devices.
Tom Nolle
Cloud in the Wild

12|4|13   |   02:23   |   15 comments


On a recent African trip I saw examples of the value of the cloud in developing nations, for educational and community development programs. We could build on this, but not only in developing economies, because these same programs are often under-supported even in first-world countries.
E2 Editors
SPONSORED: Is Malware Evading Your IPS?

11|18|13   |   03:16   |   4 comments


Intrusion prevention software is supposed to detect and block malware intrusions, but clever malware authors can evade your IPS in these five main ways.
Sara Peters
Where Have All the Mentors Gone?

9|27|13   |   3:15   |   38 comments


A good professional mentor can change your life for the better... but where do you find one?
Tom Nolle
SDN Wars & You Could Win

9|17|13   |   2:10   |   5 comments


VMware's debate with Cisco on SDN might finally create a fusion between an SDN view that's all about software and another that's all about network equipment. That would be good for every enterprise considering the cloud and SDN.
Ivan Schneider
The Future of the Smart Watch

9|12|13   |   3:19   |   39 comments


Wearing a bulky, oversized watch is good training for the next phase in wristwatches: the Internet-enabled, connected watch. Why the smartphone-tethered connected watch makes sense, plus Ivan demos an entirely new concept for the "smart watch."
Tom Nolle
Cutting Your Cloud Storage Costs

9|4|13   |   2:06   |   3 comments


Cloud storage costs are determined primarily by the rate at which files are changed and the possibility of concurrent access/update. If you can structure your storage use to optimize these factors you can cut costs, perhaps to zero.
Sara Peters
Do CIOs Need an IT Background?

8|29|13   |   2:11   |   23 comments


Most of the CIOs interviewed in the How to Become a CIO series did not start their careers as IT professionals. So is an IT background essential?
Ivan Schneider
The Internet Loves Birthdays

8|27|13   |   3:25   |   69 comments


The Internet has evolved into a machine for drumming up a chorus of "Happy Birthday" messages, from family, friends, friends of friends who you added on Facebook, random people that you circled on G+, and increasingly, automated bots. Enough already.