What I Learned... When My System Got a Major Infection

Michael Hugos, Principal, Center for Systems Innovation | 7/19/2013 | 53 comments

Michael Hugos
In my last article, I talked about how in the middle of the annual shareholders meeting, our corporate email was hit by the Sobig virus and I was in an embarrassing and dangerous situation.

In dealing with the virus that infested our email and other applications I learned it's good to be prepared, but you can't be prepared for every possible thing. Stuff happens. And when stuff happens, the process you use to respond is as important as the actual things you do. It's critical to have a simple process to guide your actions. When you need to act quickly, complex procedures inevitably break down. And a broken procedure in a high-stress situation is like losing control of a race car as it goes around a corner at 200 miles an hour.

The simple process I used has three steps, because in any situation, there are always only three things you ever need to do. First, you define what is happening and what you want to do about that (your goal). Then you design a way to get from where you are to where you need to be to accomplish your goal (your action plan). And finally, you execute the plan and build or do whatever is needed to accomplish the goal.

Afterwards, when things settle down, there is actually a fourth step, which is to review what happened and see what worked and what didn't. Then you can remember what you learned and use it the next time something similar happens.

The Sobig F virus proved to be a tricky devil. Several times that first day we did sweeps and scans of all our servers and it seemed like we had detected and erased all copies of the virus, only to have an undetected copy lurking in some obscure directory come to life again and start replicating itself wildly. By the end of the first day we realized there was no quick fix. We had to shut down our email system entirely and disable access to certain server comm ports and URLs.

We scanned and cleaned each server and only booted up the next server after we knew the previous one was entirely clean. We had to do this to all the servers running internal business applications as well. That took much of the second day. And all the while we communicated openly with our business units and our customers and suppliers to let them know what was happening and help them out when they found some of their servers were infected from emails we sent out.

As we tackled this problem, we were clear about our goal: Get things back to normal. But we went through the design and build steps several times. Instead of getting stuck on analysis, we tried things and we learned. We communicated with people outside the company and got useful advice. We kept updating our solution design as we progressed. We stayed focused and worked in shifts around the clock. By the middle of the second day we had the situation under control, and by the morning of the third day all systems and email were back up and performing as they should.

After we restored the system, we also had to restore our reputation with the company and the shareholders. We were able to do that quicker than I feared, because I owned up to what was happening right away and did not waste time with excuses that nobody wanted to hear anyway. We were clear about what we were going to do to fix the problem, and we did fix the problem over the next two days as we said we would. People know the IT world is a wild ride these days. They cut us some slack on this breach because it was the first breach of that magnitude, and we learned from it and put new procedures in place. Nothing like it happened again. If there had been another breach of that magnitude it would have been a reason for my dismissal. I took the experience seriously.

In our review after things settled down, we identified some lessons learned and used them going forward. They seemed mostly pretty obvious afterwards, but we hadn't seen them or heeded them before the breach. We learned to isolate our key internal application systems and their databases from easy access by email or other internal systems. We started using temporary data files or central data warehouses to move data between systems and scan the heck out of that data before letting it move between systems. We also tightened up email protocols and started blocking suspicious attachments.

Yet after all was said and done, we knew we could never put enough technology and procedures in place to cover all potential threats. It was the power of a simple, effective, problem-solving process that everyone understood and everyone used that got us through the crisis. Knowing the drill, keeping it current, and doing it well when it counts is the ultimate weapon against the unexpected stuff that can happen at any moment in IT operations.

Related posts:

View Comments: Newest First | Oldest First | Threaded View
Page 1 / 6   >   >>
stotheco   What I Learned... When My System Got a Major Infection   8/2/2013 4:00:49 AM
Re: pinpointing
Great, do keep us updated. I don't think there's anything wrong with being passionate about something you believe in...in fact, I think that is a very good thing. We need more passionate people around to advocate their causes and to help others in the same way.
stotheco   What I Learned... When My System Got a Major Infection   8/2/2013 3:59:59 AM
Re: Re : What I Learned... When My System Got a Major Infection
Back up, back up, back up! This is maybe the most basic of advice that IT can give you. If you don't follow it and if something does happen, then all your data is a goner. Don't learn this lesson the hard way, like so many others have had to.
Susan Nunziata   What I Learned... When My System Got a Major Infection   7/30/2013 2:00:50 PM
Re: pinpointing
@Ashish: I know what you mean, I am the same way when I'm doing any public speaking on topics I'm passionate about.

Your enthusiasm is usually contagious, though, so that is a good thing for the classes that you are lecturing for. 

Nomi   What I Learned... When My System Got a Major Infection   7/30/2013 5:26:55 AM
Re: pinpointing
Susan I agree with you. The wealth of information is enormous and we all want to have that with us.
eethtworkz   What I Learned... When My System Got a Major Infection   7/30/2013 4:58:22 AM
Re: pinpointing

I will have to check up on that issue!

Will follow-through on this and keep the Community Updated here.

It was a favor for a Friend's Friend-Since he knew how passionate I am about IT Security;he plugged me into to do a Guest Lecture(sorta like a Visiting Industry Faculty).

I have another big problem-When I get involved in working or speaking on some issue I am passionate about;I don't know where to Stop.


Can create major issues with Time limits,etc.



Susan Nunziata   What I Learned... When My System Got a Major Infection   7/29/2013 5:45:53 PM
Re: pinpointing
@Ashish: Thank you! for sharing this incredible wealth of information.

Have your lectures been recorded? I bet the community here would love to see some of them. 
SunitaT   What I Learned... When My System Got a Major Infection   7/29/2013 5:16:52 AM
Re : What I Learned... When My System Got a Major Infection
This is really a good lesson learned. In case of hard disk crash of office PC, this is always necessary to keep the data in network or take back-up in cd.
eethtworkz   What I Learned... When My System Got a Major Infection   7/29/2013 1:43:01 AM
Re: pinpointing

No Issues.

The Competition in the MDM Space today,especially for Android Smartphones is Frankly Speaking not upto the Mark.

And No I Don't blame the MDM Vendors for this issue.

The problem is there is a lack of consistency between various flavors and Forks of Android today.

So nobody really knows who is doing what and how to protect it.

You will like this Excerpt from a Recent Lecture I gave at a University on cloud Security recently-


There is no Fixed percentage as such which says Cloud Security is more/less Vulnerable than In-house/On-premise solutions.

It all depends on how fast you patch your systems in case of Vulnerabilities and how responsible the Vendor is in Maintaining Patching cycles.

For instance Apple,Oracle,Facebook and Adobe are terrible at Fixing patches while on the other end of the Spectrum Microsoft,IBM,SAP,Mozilla ,LinkedIn and Google are Generally very Good at Fixing Patches as soon as a Vulnerability is Discovered.

But then with On-premise systems there is added responsibility on the heads of IT Administrators to ensure their systems are patched ontime;while in Cloud this responsibility lies solely with the Vendors.

Is it easy/Difficult to penetrate security today?

Again that depends.While basic Sandboxing has improved Security of most Systems;Plugins at the Browser level are the source of most Vulnerabilities today[Not at the OS Level anymore on most of the Latest OSes around].

The only exception here is Android for Smartphones.Simply because there are so many Different Forks and Flavors floating around(and each Vendor tends to mould the Flavor of Android according to their specific needs patching becomes a veritable nightmare here on most Android Systems-How can you maintain a Common MDM Patching Solution when you don't even know which Fork is going where????].But if you restrict yourself only to the Top Vendor in this space-Samsung then I can say they have done a reasonably good job of collaborating/cooperating with Google here and ensuring that most of Security Issues are actually at the App level rather than the OS Level.

With the rest?Karbonn/Micromax/LG/Sony/HTC/Amazon/Huawei??? Nobody really knows.

Frankly Speaking the whole Android Ecosystem is a Total Mess when it comes to Security Issues.Security Pros want a consistency of approach to ensure we can apply the Same Level of Policies everywhere;with the way the Android Ecosystem has developed today that is not the case.

This was the Primary reason why Google took back most of the Responsibility for the Android OS back in house last year.

Now since responsibility for Patching these plugins/Apps  lies  not with the Browser/OS  vendor( but usually with a 3rd party who makes the Plug-ins) it has added considerable levels of Complexity to the whole procedure.
Who takes responsibility to ensure your Plug-ins are patched?

Mozilla has now implemented great Browser Plug-in protection for all the most Commonly used Plug-ins where if you aren't using the most recently patched plug-in it warns you immediately and tells you use the Plug-in entirely at your Personal responsibility.

All others are far behind in this space.

You can get more information from My friend Grimes in his post here


and from my Analyst friends at NSS Labs here

What happens is most IT Managers(even CIOs) have neither the Training in Security nor do They have any understanding of Security and how it has evolved today.So they readily give in to most Demands(however unreasonable they are from the Client side).

I am very different that my Training was in Cryptography(from Columbia no less) and I still do a lot of Research in this space today;So whenever anyone makes any Nonsense  on this issue -I am able to Shoot them down very,very easily.
Susan Nunziata   What I Learned... When My System Got a Major Infection   7/28/2013 9:18:00 PM
Re: pinpointing
@Ashish: Ah, thank you for explaining that. The misunderstanding was mine. 

I still applaud your company for keeping its policies consistent across the board, regardless of executive title or rank on the corporate ladder.

It will be interesting to watch in the next few years as competition in the MDM sector continues to see if there are changes to pricing models that enable more companies to make use of these tools.
shakeeb   What I Learned... When My System Got a Major Infection   7/27/2013 2:30:55 AM
Re: steps
I also think continuous monitoring is also an important aspect. This will allow the system administrators to monitor the unusual behaviors in the system.
Page 1 / 6   >   >>

The blogs and comments posted on EnterpriseEfficiency.com do not reflect the views of TechWeb, EnterpriseEfficiency.com, or its sponsors. EnterpriseEfficiency.com, TechWeb, and its sponsors do not assume responsibility for any comments, claims, or opinions made by authors and bloggers. They are no substitute for your own research and should not be relied upon for trading or any other purpose.

More Blogs from Michael Hugos
Michael Hugos   3/11/2014   9 comments
CIOs and COOs generally play a more supporting role than their more glamorous counterparts in marketing, sales, and finance. You can turn that to an advantage if you want to build a better ...
Michael Hugos   3/5/2014   15 comments
CFOs can appear to be a smug and self-assured bunch, but underneath that smug exterior there lurks much uncertainty and doubt.
Michael Hugos   2/25/2014   26 comments
In our continuing series about whom the CIO should make friends with, the next most pressured person in your company (behind the CMO) is probably the VP of Sales. There might be ways to ...
Michael Hugos   2/19/2014   21 comments
If you are a CIO who wants to ensure your place in the organization, a good place to start is with the CMO. That is because the CMO is most likely the C-suite executive under the most ...
Michael Hugos   2/5/2014   30 comments
At present we are living in the twilight of the Industrial Age from the last century, just as people in 1914 were living in the twilight of the Victorian Age from the century before that. ...
Latest Archived Broadcast
We talk with Bernard Golden about accelerating application delivery in the cloud.
On-demand Video with Chat
Register for this video discussion to learn how tablets can provide true business usability and productivity.
E2 IT Migration Zones
IT Migration Zone - UK
Why PowerShell Is Important
Reduce the Windows 8 Footprint for VDI
Rethinking Storage Management
IT Migration Zone - FR
SQL Server : 240 To de mémoire flash pour votre data warehouse
Quand Office vient booster les revenus Cloud et Android de Microsoft
Windows Phone : Nokia veut davantage d'applications (et les utilisateurs aussi)
IT Migration Zone - DE
Cloud Computing: Warum Unternehmen trotz NSA auf die „private“ Wolke setzen sollten
Cloud Computing bleibt Wachstumsmarkt – Windows Azure ist Vorreiter
Like Us on Facebook
Twitter Feed
Enterprise Efficiency Twitter Feed
Site Moderators Wanted
Enterprise Efficiency is looking for engaged readers to moderate the message boards on this site. Engage in high-IQ conversations with IT industry leaders; earn kudos and perks. Interested? E-mail:
[email protected]
Dell's Efficiency Modeling Tool
The major problem facing the CIO is how to measure the effectiveness of the IT department. Learn how Dell’s Efficiency Modeling Tool gives the CIO two clear, powerful numbers: Efficiency Quotient and Impact Quotient. These numbers can be transforma¬tive not only to the department, but to the entire enterprise.

Read the full report
The State of Enterprise Efficiency in the Virtual Era: Virtualization – Smart Approaches to Maximize Gains
Virtualization is a presence in nearly all enterprise data centers. But not all companies are using it to its best effect. Learn the common characteristics of success, what barriers companies face, and how to get the most from your efforts.

Read the full report
Informed CIO: Dollars & Sense: Virtual Desktop Infrastructure
Cut through the VDI hype and get the full picture -- including ROI and the impact on your Data Center -- to make an informed decision about your virtual desktop infrastructure deployments.

Read the full report
A Video Case Study – Translational Genomics Research Institute
e2 Video

On the Case
TGen IT: Where We're Going Next

7|11|12   |   08:12   |   10 comments

Now that TGen has broken new ground in genomic research by using Dell's storage, cloud, and high-performance computing solutions, the company discusses what will come next for it and for personalized medicine.
On the Case
Better Care Through Better Communications

6|6|12   |   02:24   |   11 comments

The achievements of the TGen/Dell project could improve how all people receive healthcare, because they are creating ways to improve end-to-end communication of medical data.
On the Case
TGen IT: Where We Are Now

5|15|12   |   06:58   |   6 comments

TGen is breaking new ground in genomic research by using Dell's storage, cloud, and high-performance computing solutions.
On the Case
TGen IT: Where We Were

4|27|12   |   06:45   |   10 comments

The Translational Genomics Research Institute wanted to save lives, but its efforts were hobbled by immense computing challenges related to collecting, processing, sharing, and storing enormous amounts of data.
On the Case
1,200% Faster

4|18|12   |   02:27   |   12 comments

Through their partnership, Dell and TGen have increased the speed of TGen’s medical research by 1,200 percent.
On the Case
IT May Improve Children's Chances of Survival

4|17|12   |   02:12   |   8 comments

IT is helping medical researchers reach breakthroughs in a way and pace never seen before.
On the Case
Medical Advances in the Cloud

4|10|12   |   1:25   |   5 comments

TGen and Dell are pushing the boundaries of computing, and harnessing the power of the cloud to improve healthcare.
On the Case
TGen: Living the Mission

4|9|12   |   2:25   |   3 comments

TGen's CIO puts the organizational mission at the heart of everything the IT staff does.
On the Case
TGen Speeding Up Biomedical Research to Save More Lives

4|5|12   |   1:59   |   6 comments

The Translational Genomics Research Institute is revamping its computing to improve speed, storage, and collaboration – and, most importantly, to save lives.
On the Case
Computing Power Helping to Save Children's Lives

3|28|12   |   2:13   |   3 comments

The Translational Genomics Institute’s partnership with Dell is enabling them to treat kids with neuroblastoma more quickly and save more lives.
Tom Nolle
The Big Reason to Use Office

3|18|14   |   02:24   |   46 comments

Office and personal productivity tools come in a first-class and coach flavor set, but what makes the difference is primarily little things that most users won't encounter. What's the big issue in using something other than Office, and can you get around it?
E2 Editors
SPONSORED: Mobile Security — A Use Case

3|4|14   |   04:27   |   16 comments

New mobile security solutions can accommodate a wide array of needs, including those of a complex university environment.
Tom Nolle
Killing Net Neutrality Might Save You Money

1|16|14   |   2:13   |   16 comments

The DC Court of Appeals voided most of the Neutrality Order, and whatever it might mean for the Internet overall, it might mean better and cheaper Internet VPNs for businesses.
Tom Nolle
The Internet of Everythinguseful

1|10|14   |   2:18   |   19 comments

We really don't want an "Internet of Everything" but even building an Internet of Everythinguseful means setting some ground rules to insure there's value in the process and that costs and risks are minimized.
Tom Nolle
Maturing Google Chrome

12|30|13   |   2.18   |   25 comments

Google's Chrome OS has a lot of potential value and a lot of recent press, but it still needs something to make it more than a thin client. It needs cloud integration, it needs extended APIs via web services, and it needs to suck it up and support a hard drive.
Sara Peters
No More Cookie-Cutter IT

12|23|13   |   03.58   |   21 comments

Creating the right combination of technology, people, and processes for your IT organization is a lot like baking Christmas cookies.
Sara Peters
Smart Wigs Not a Smart Idea

12|5|13   |   3:01   |   46 comments

Sony is seeking a patent for wigs that contain computing devices.
Tom Nolle
Cloud in the Wild

12|4|13   |   02:23   |   15 comments

On a recent African trip I saw examples of the value of the cloud in developing nations, for educational and community development programs. We could build on this, but not only in developing economies, because these same programs are often under-supported even in first-world countries.
E2 Editors
SPONSORED: Is Malware Evading Your IPS?

11|18|13   |   03:16   |   4 comments

Intrusion prevention software is supposed to detect and block malware intrusions, but clever malware authors can evade your IPS in these five main ways.
Sara Peters
Where Have All the Mentors Gone?

9|27|13   |   3:15   |   38 comments

A good professional mentor can change your life for the better... but where do you find one?
Tom Nolle
SDN Wars & You Could Win

9|17|13   |   2:10   |   5 comments

VMware's debate with Cisco on SDN might finally create a fusion between an SDN view that's all about software and another that's all about network equipment. That would be good for every enterprise considering the cloud and SDN.
Ivan Schneider
The Future of the Smart Watch

9|12|13   |   3:19   |   39 comments

Wearing a bulky, oversized watch is good training for the next phase in wristwatches: the Internet-enabled, connected watch. Why the smartphone-tethered connected watch makes sense, plus Ivan demos an entirely new concept for the "smart watch."
Tom Nolle
Cutting Your Cloud Storage Costs

9|4|13   |   2:06   |   3 comments

Cloud storage costs are determined primarily by the rate at which files are changed and the possibility of concurrent access/update. If you can structure your storage use to optimize these factors you can cut costs, perhaps to zero.
Sara Peters
Do CIOs Need an IT Background?

8|29|13   |   2:11   |   23 comments

Most of the CIOs interviewed in the How to Become a CIO series did not start their careers as IT professionals. So is an IT background essential?
Ivan Schneider
The Internet Loves Birthdays

8|27|13   |   3:25   |   69 comments

The Internet has evolved into a machine for drumming up a chorus of "Happy Birthday" messages, from family, friends, friends of friends who you added on Facebook, random people that you circled on G+, and increasingly, automated bots. Enough already.