[Editor's Note: This is part of a new series written by CIOs discussing their thought process and lessons learned from major events in their tenure as CIO. Tomorrow we will print a companion blog to this one on lessons learned after this event. We will print similar stories on this and other topics by many CIOs who write for E2.]
Major outages are to CIO tenure what kryptonite is to Superman. This was especially true during our last outage, which happened during my second year at the USTA during the first week of the US Open. We generate approximately 85 percent of our annual revenue during the US Open, so for us it is the equivalent of 14 straight days of Black Friday to a retailer.
Unfortunately, we encountered a virus that negatively impacted network availability for parts of almost two days during the first week of the main draw. Here's what I was thinking at that time:
1. I need to drive a total sense of urgency and get all hands on deck to resolve this problem. We needed to find additional outside expertise to help us unravel the root cause behind this issue. At the time we had a relationship with an outsource partner who clearly did not have the expertise or acumen to help us resolve the issue. Furthermore, the organization had never invested in the technology infrastructure required to support an event the magnitude of the US Open. So we were flying blind with antiquated technology with a clueless vendor during our most important event.
2. I had to deal with the fallout of a major issue during our prize jewel with a management team that already felt IT was a weak link in the chain and a financial drain on the organization. I had to convey bad news, without any answers as to what had happened, why, what we were going to be able to do about it, or an ETA as to when the issue would be resolved. I needed to do this in a way that conveyed a sense of urgency while concurrently exhibiting a level of calm and composure to assure people that we were on top of the situation and that we would get things under control. As the old antiperspirant commercial tagline went "never let them see you sweat."
3. I was thinking that I had to find a way to not let this unfortunate but isolated issue derail our transformation efforts and erase all the momentum and progress we had made in the first year of our program to overhaul IT. All it takes is one major outage to allow doubt to creep back into people's minds. People quickly forget all the good things you've done when one really bad thing happens. I lovingly refer to this as the "Janet Jackson School of Management" -- what have you done for me lately?
4. Of course, I was also thinking that the average tenure of a CIO at that time was about three years, and that I was likely to fall 18 months short of this metric. I had to summon up all the emotional discipline I could muster to not allow my head to go to that "Oh sh-t" place where you feel that your journey has come to the end of the road. There was no value in allowing myself to go to what my dear friend Carol Zierhoffer, the global CIO of Xerox, refers to as your "emotional basement," nor could I allow my team to smell fear on me and be distracted from the issue at hand.
With all of those thoughts in mind, I went about addressing the outage. In a blog tomorrow, come back to see how we got out of the problem, what we learned, and whether I still had a job.