What I Was Thinking... During Our Last Outage

Larry Bonfante, Founder, CIO Bench Coach | 4/17/2013 | 17 comments

Larry Bonfante
[Editor's Note: This is part of a new series written by CIOs discussing their thought process and lessons learned from major events in their tenure as CIO. Tomorrow we will print a companion blog to this one on lessons learned after this event. We will print similar stories on this and other topics by many CIOs who write for E2.]

Major outages are to CIO tenure what kryptonite is to Superman. This was especially true during our last outage, which happened during my second year at the USTA during the first week of the US Open. We generate approximately 85 percent of our annual revenue during the US Open, so for us it is the equivalent of 14 straight days of Black Friday to a retailer.

Unfortunately, we encountered a virus that negatively impacted network availability for parts of almost two days during the first week of the main draw. Here's what I was thinking at that time:

1. I need to drive a total sense of urgency and get all hands on deck to resolve this problem. We needed to find additional outside expertise to help us unravel the root cause behind this issue. At the time we had a relationship with an outsource partner who clearly did not have the expertise or acumen to help us resolve the issue. Furthermore, the organization had never invested in the technology infrastructure required to support an event the magnitude of the US Open. So we were flying blind with antiquated technology with a clueless vendor during our most important event.

2. I had to deal with the fallout of a major issue during our prize jewel with a management team that already felt IT was a weak link in the chain and a financial drain on the organization. I had to convey bad news, without any answers as to what had happened, why, what we were going to be able to do about it, or an ETA as to when the issue would be resolved. I needed to do this in a way that conveyed a sense of urgency while concurrently exhibiting a level of calm and composure to assure people that we were on top of the situation and that we would get things under control. As the old antiperspirant commercial tagline went "never let them see you sweat."

3. I was thinking that I had to find a way to not let this unfortunate but isolated issue derail our transformation efforts and erase all the momentum and progress we had made in the first year of our program to overhaul IT. All it takes is one major outage to allow doubt to creep back into people's minds. People quickly forget all the good things you've done when one really bad thing happens. I lovingly refer to this as the "Janet Jackson School of Management" -- what have you done for me lately?

4. Of course, I was also thinking that the average tenure of a CIO at that time was about three years, and that I was likely to fall 18 months short of this metric. I had to summon up all the emotional discipline I could muster to not allow my head to go to that "Oh sh-t" place where you feel that your journey has come to the end of the road. There was no value in allowing myself to go to what my dear friend Carol Zierhoffer, the global CIO of Xerox, refers to as your "emotional basement," nor could I allow my team to smell fear on me and be distracted from the issue at hand.

With all of those thoughts in mind, I went about addressing the outage. In a blog tomorrow, come back to see how we got out of the problem, what we learned, and whether I still had a job.

View Comments: Newest First | Oldest First | Threaded View
Page 1 / 2   >   >>
SaneIT   What I Was Thinking... During Our Last Outage   4/23/2013 7:15:28 AM
Re: Stop Button
I can se why it would work, first off the tangible ticket that is handed to someone.  Putting it in their hands literally is a different feeling than seeing it pop up into the helpdesk queue.  Then you have the mess that is most helpdesk databases with ancient tickets that just sit there open and cluttering up the place.  It's kind of like a fresh start every time the power goes out when you switch to your paper plan.
vnewman   What I Was Thinking... During Our Last Outage   4/22/2013 8:45:52 PM
Re: Stop Button
@SaneIT - it wasn't an easy sell - I will tell you that much...but we all got used to it and believe it or not...I hate to say this...sometimes it seems to work better than the computerized tracking system. 

Maybe it's because it's something different.  Maybe because it's an exception to the rule.  Maybe because we only have to do it for a short time.  Or maybe it's because the open cases sometimes get lost in a sea of emails we all get.  Regardless - it came out of necessity after getting chewed out one too many times when the system was down and hearing, "So-and-so said they called the help desk and reported X and said no one ever got back to them."

Johnny Carson was on to something :)
singlemud   What I Was Thinking... During Our Last Outage   4/22/2013 2:16:11 PM
thanks a lot for your sharing
It is a great post, and many practicial advice,  "never let them see you sweat." I like this in the context of CIO facing outrage.
SaneIT   What I Was Thinking... During Our Last Outage   4/22/2013 8:06:08 AM
Re: Stop Button
@vnewman, that's in interesting take on how to do helpdesk tickets when even the techs lose connectivity.  I have trouble getting people to open tickets even when the network is up, I can imagine their faces when I tell them they have to hand write a ticket twice....
Zaius   What I Was Thinking... During Our Last Outage   4/22/2013 1:21:59 AM
Re: Stop Button
Well said: "The show must go on".

At the moment of disaster, there are two things needed: one team to get it restored, and one to keep the tasks going. The second can be managed nicely if we are well armed with a plan of action. Often we just think about recovery, and forget that there is the need to keeo things running.
MDMConsult   What I Was Thinking... During Our Last Outage   4/20/2013 11:05:59 AM
Re: Stop Button
Customer retention due to unreliability issues could best be managed by maintaining revenue at low cost and increasing customer lifetime value metrics. Customer satisfaction in reliability is highly important if unreliability negatively affects the consumer. 
vnewman   What I Was Thinking... During Our Last Outage   4/19/2013 9:38:11 AM
Re: Stop Button
It's always nice to have a "how did we operate before computers" contingency plan for the most integral functions of the firm's business.  In this case, the draw still needs to be made, the schedule of play must be coordinated, and the players must have transportation.  All of these things can happen manually, although I'm sure it takes longer to communicate with all the parties involved without a network.  But having an "old fashioned" way to conduct business is critical for getting the most basic tasks completed so you don't suffer a complete work stoppage.

As an example, in our department, if the network goes down (hasn't in a long time), we resort to the "tonight show method" of keeping track of incoming help desk calls.  We assign the task by giving the tech a card and a duplicate goes up on a white board in the help desk office until it is completed.  Apparently this is how Johnny Carson would schedule his guests each night...and as the saying goes - the show must go on...
SaneIT   What I Was Thinking... During Our Last Outage   4/18/2013 8:14:26 AM
Re: Stop Button
How true, everyone gets wound up and there is a lot of energy spent just fretting over the impact.  Another issue is that these things tend to happen at the worst possible time.  People get into crunch mode and miss the little things like updating virus databases and by the time it is noticed it's crippling an already overloaded network and overloaded employees.  No one has time to fix it, no one has time to be away from their PC while it is fixed but no one has time to deal with their computer crashing every half hour either.
Zaius   What I Was Thinking... During Our Last Outage   4/17/2013 8:02:01 PM
Re: Stop Button
These kind of events not only terrorizes the CIO; it puts everyone in a 'adrenaline' situation. IT people also fear that CIO going to be pretty unhappy and many of theri future prospects are ruined.

Anyway, looking forward to the next installment.
kstaron   What I Was Thinking... During Our Last Outage   4/17/2013 5:59:09 PM
Stop Button
Yikes! Talk about a bad time for things to go south. I agree that if you can hit the stop button on the elevator as you feel yourself plummeting to your emotional basement you can accomplish alot more. Cool, calm, collected always beats cranky, frustrated, and frazzled.
Page 1 / 2   >   >>

The blogs and comments posted on EnterpriseEfficiency.com do not reflect the views of TechWeb, EnterpriseEfficiency.com, or its sponsors. EnterpriseEfficiency.com, TechWeb, and its sponsors do not assume responsibility for any comments, claims, or opinions made by authors and bloggers. They are no substitute for your own research and should not be relied upon for trading or any other purpose.

More Blogs from Larry Bonfante
Larry Bonfante   5/7/2014   58 comments
Most leaders feel a sense of responsibility for creating a professional work environment where their people can succeed, but I feel this is nowhere near enough.
Larry Bonfante   4/28/2014   24 comments
Of all the competencies required to be an effective CIO, perhaps none is more important than being able to build positive relationships with your key stakeholders. I've never seen a CIO ...
Larry Bonfante   4/9/2014   10 comments
When every capital expenditure is put under a microscope, it's harder than ever to continue to make the necessary investments in refreshing the technology our companies need to compete in ...
Larry Bonfante   3/25/2014   20 comments
Today we are flooded with an almost infinite number of data streams. From traditional sources such as registration and consumer purchasing platforms to social media feeds, tweets, and ...
Larry Bonfante   2/25/2014   19 comments
CIOs don't work alone. Effective CIOs develop effective partnerships with the other executives in the "C-Suite" -- partnerships that take time, effort, and understanding to build.
Latest Archived Broadcast
We talk with Bernard Golden about accelerating application delivery in the cloud.
On-demand Video with Chat
Register for this video discussion to learn how tablets can provide true business usability and productivity.
E2 IT Migration Zones
IT Migration Zone - UK
Why PowerShell Is Important
Reduce the Windows 8 Footprint for VDI
Rethinking Storage Management
IT Migration Zone - FR
SQL Server : 240 To de mémoire flash pour votre data warehouse
Quand Office vient booster les revenus Cloud et Android de Microsoft
Windows Phone : Nokia veut davantage d'applications (et les utilisateurs aussi)
IT Migration Zone - DE
Cloud Computing: Warum Unternehmen trotz NSA auf die „private“ Wolke setzen sollten
Cloud Computing bleibt Wachstumsmarkt – Windows Azure ist Vorreiter
Like Us on Facebook
Twitter Feed
Enterprise Efficiency Twitter Feed
Site Moderators Wanted
Enterprise Efficiency is looking for engaged readers to moderate the message boards on this site. Engage in high-IQ conversations with IT industry leaders; earn kudos and perks. Interested? E-mail:
Dell's Efficiency Modeling Tool
The major problem facing the CIO is how to measure the effectiveness of the IT department. Learn how Dell’s Efficiency Modeling Tool gives the CIO two clear, powerful numbers: Efficiency Quotient and Impact Quotient. These numbers can be transforma¬tive not only to the department, but to the entire enterprise.

Read the full report
The State of Enterprise Efficiency in the Virtual Era: Virtualization – Smart Approaches to Maximize Gains
Virtualization is a presence in nearly all enterprise data centers. But not all companies are using it to its best effect. Learn the common characteristics of success, what barriers companies face, and how to get the most from your efforts.

Read the full report
Informed CIO: Dollars & Sense: Virtual Desktop Infrastructure
Cut through the VDI hype and get the full picture -- including ROI and the impact on your Data Center -- to make an informed decision about your virtual desktop infrastructure deployments.

Read the full report
A Video Case Study – Translational Genomics Research Institute
e2 Video

On the Case
TGen IT: Where We're Going Next

7|11|12   |   08:12   |   10 comments

Now that TGen has broken new ground in genomic research by using Dell's storage, cloud, and high-performance computing solutions, the company discusses what will come next for it and for personalized medicine.
On the Case
Better Care Through Better Communications

6|6|12   |   02:24   |   11 comments

The achievements of the TGen/Dell project could improve how all people receive healthcare, because they are creating ways to improve end-to-end communication of medical data.
On the Case
TGen IT: Where We Are Now

5|15|12   |   06:58   |   6 comments

TGen is breaking new ground in genomic research by using Dell's storage, cloud, and high-performance computing solutions.
On the Case
TGen IT: Where We Were

4|27|12   |   06:45   |   10 comments

The Translational Genomics Research Institute wanted to save lives, but its efforts were hobbled by immense computing challenges related to collecting, processing, sharing, and storing enormous amounts of data.
On the Case
1,200% Faster

4|18|12   |   02:27   |   12 comments

Through their partnership, Dell and TGen have increased the speed of TGen’s medical research by 1,200 percent.
On the Case
IT May Improve Children's Chances of Survival

4|17|12   |   02:12   |   8 comments

IT is helping medical researchers reach breakthroughs in a way and pace never seen before.
On the Case
Medical Advances in the Cloud

4|10|12   |   1:25   |   5 comments

TGen and Dell are pushing the boundaries of computing, and harnessing the power of the cloud to improve healthcare.
On the Case
TGen: Living the Mission

4|9|12   |   2:25   |   3 comments

TGen's CIO puts the organizational mission at the heart of everything the IT staff does.
On the Case
TGen Speeding Up Biomedical Research to Save More Lives

4|5|12   |   1:59   |   6 comments

The Translational Genomics Research Institute is revamping its computing to improve speed, storage, and collaboration – and, most importantly, to save lives.
On the Case
Computing Power Helping to Save Children's Lives

3|28|12   |   2:13   |   3 comments

The Translational Genomics Institute’s partnership with Dell is enabling them to treat kids with neuroblastoma more quickly and save more lives.
Tom Nolle
The Big Reason to Use Office

3|18|14   |   02:24   |   46 comments

Office and personal productivity tools come in a first-class and coach flavor set, but what makes the difference is primarily little things that most users won't encounter. What's the big issue in using something other than Office, and can you get around it?
E2 Editors
SPONSORED: Mobile Security — A Use Case

3|4|14   |   04:27   |   16 comments

New mobile security solutions can accommodate a wide array of needs, including those of a complex university environment.
Tom Nolle
Killing Net Neutrality Might Save You Money

1|16|14   |   2:13   |   16 comments

The DC Court of Appeals voided most of the Neutrality Order, and whatever it might mean for the Internet overall, it might mean better and cheaper Internet VPNs for businesses.
Tom Nolle
The Internet of Everythinguseful

1|10|14   |   2:18   |   19 comments

We really don't want an "Internet of Everything" but even building an Internet of Everythinguseful means setting some ground rules to insure there's value in the process and that costs and risks are minimized.
Tom Nolle
Maturing Google Chrome

12|30|13   |   2.18   |   25 comments

Google's Chrome OS has a lot of potential value and a lot of recent press, but it still needs something to make it more than a thin client. It needs cloud integration, it needs extended APIs via web services, and it needs to suck it up and support a hard drive.
Sara Peters
No More Cookie-Cutter IT

12|23|13   |   03.58   |   21 comments

Creating the right combination of technology, people, and processes for your IT organization is a lot like baking Christmas cookies.
Sara Peters
Smart Wigs Not a Smart Idea

12|5|13   |   3:01   |   46 comments

Sony is seeking a patent for wigs that contain computing devices.
Tom Nolle
Cloud in the Wild

12|4|13   |   02:23   |   15 comments

On a recent African trip I saw examples of the value of the cloud in developing nations, for educational and community development programs. We could build on this, but not only in developing economies, because these same programs are often under-supported even in first-world countries.
E2 Editors
SPONSORED: Is Malware Evading Your IPS?

11|18|13   |   03:16   |   4 comments

Intrusion prevention software is supposed to detect and block malware intrusions, but clever malware authors can evade your IPS in these five main ways.
Sara Peters
Where Have All the Mentors Gone?

9|27|13   |   3:15   |   38 comments

A good professional mentor can change your life for the better... but where do you find one?
Tom Nolle
SDN Wars & You Could Win

9|17|13   |   2:10   |   5 comments

VMware's debate with Cisco on SDN might finally create a fusion between an SDN view that's all about software and another that's all about network equipment. That would be good for every enterprise considering the cloud and SDN.
Ivan Schneider
The Future of the Smart Watch

9|12|13   |   3:19   |   39 comments

Wearing a bulky, oversized watch is good training for the next phase in wristwatches: the Internet-enabled, connected watch. Why the smartphone-tethered connected watch makes sense, plus Ivan demos an entirely new concept for the "smart watch."
Tom Nolle
Cutting Your Cloud Storage Costs

9|4|13   |   2:06   |   3 comments

Cloud storage costs are determined primarily by the rate at which files are changed and the possibility of concurrent access/update. If you can structure your storage use to optimize these factors you can cut costs, perhaps to zero.
Sara Peters
Do CIOs Need an IT Background?

8|29|13   |   2:11   |   23 comments

Most of the CIOs interviewed in the How to Become a CIO series did not start their careers as IT professionals. So is an IT background essential?
Ivan Schneider
The Internet Loves Birthdays

8|27|13   |   3:25   |   69 comments

The Internet has evolved into a machine for drumming up a chorus of "Happy Birthday" messages, from family, friends, friends of friends who you added on Facebook, random people that you circled on G+, and increasingly, automated bots. Enough already.