A few months ago, the E2 editors asked CIOs to blog about various experiences (mostly bad) and what we had learned from them. As such, I decided to take the bait and write about the worst mistake I ever made as a CIO -- and what I learned from it.
It came shortly after I was hired as the CIO of an organization in the late spring of 1999. The organization had not even begun preparing for the infamous Y2K bug. I had recently completed a high-profile Y2K preparation for another organization, and I was put in charge of this organization's IT department primarily because of my experience with Y2K readiness.
One of the first things I did was perform a Y2K readiness assessment. The results were not promising. Across the organization, there were a couple hundred people who were still using 486 PCs, which were not Y2K compliant. The problem was that my IT budget did not allow me to go in and replace all these PCs. I had to find a way of making those computers Y2K compliant.
While researching the problem, I found a company that created modified processors that would fit into a 486 socket and trick the software into thinking it was running on a Pentium-class computer. I talked the vendor into sending me a sample of the chip. I installed it into an old machine I found in the closet, did some basic lab testing, and confirmed that the chip was indeed Y2K compliant. That being the case, I placed a bulk order for the chips.
This is where things went horribly wrong. Even though the modified processors truly were Y2K compliant, they ended up being slower than the native 486 processors. I hadn't done an extensive amount of testing, because time was of the essence. I was under a tremendous amount of pressure to come up with a solution and implement it quickly. Because of the time crunch, I had neglected to do a pilot deployment. Had I done that, I would have quickly determined that, even though the chips worked fine in a lab environment, they simply weren't up to the job of handling the user's normal workload.
Ultimately, it was nothing but dumb luck that kept my huge mistake from turning into a major catastrophe. One of the facilities I oversaw happened to be replacing all its computers. The old computers really weren't that old, so I was able to use them to replace all the 486 machines throughout the organization.
Obviously I screwed up big time by not performing a pilot deployment. It's an embarrassing mistake, and I seriously considered pretending it never happened, even though the incident occurred nearly 15 years ago. Even so, the incident did teach me several valuable lessons:
- There is no substitute for a pilot deployment program. Sometimes lab testing is simply inadequate.
- Using a modified processor was a bad idea. Even if it had performed flawlessly, I shouldn't have used it, because the Microsoft Hardware Compatibility List did not certify Windows to work with the processor. Though no compatibility issues were found, there were no guarantees. By using a modified processor, I essentially put the desktop computers into an unsupported state.
- Even though my budget did not allow me to replace the computers that really needed to be replaced, there were solutions less drastic than using a modified processor (though those solutions were not quite as cheap). I could have replaced the system board, processor, and memory in each of the aging machines. This would have allowed me to reuse the computer case, power supply, hard disk, and cabling. Doing so would have cost more than using modified processors, but it still would have been far cheaper than replacing entire PCs.
So there you have it. That is by far the worst mistake I ever made while I was working as a CIO, and those are the hard lessons I learned from it. I ultimately recovered from the mistake and had a long career until I recently decided to become a consultant and a freelancer. Mistakes are part of any job, of course, but since I have shared such a personal and embarrassing story, I ask that you please be kind with your comments. Maybe you can share a mistake of your own and what can be learned from it.