IT Lessons From a Tragedy

Curtis Franklin Jr., Executive Editor | 4/16/2013 | 43 comments

Curtis Franklin Jr.
Yesterday's bombing in Boston was horrible for those caught by the blast and terrible for their friends and families. The impact reached far beyond the bomb's blast radius.

As people across the country (and around the world) sought to find out more about the event, they went to media websites for reports on the known and unfolding, and sometimes found no response to their queries. Overnight analysis of test results has shed some light on exactly how significant the impact on the websites became -- and what companies faced with similar spikes in demand might do to keep data flowing.

Keynote Systems performs ongoing monitoring of a wide variety of websites. Among their regular reports is a weekly look at how the websites of 22 different media organizations perform. In response to yesterday's events, they released an overnight report on how those sites had handled the high demand. The company sent Enterprise Efficiency a copy of the report and I had a chance to talk with one of their performance experts. What I got was an interesting picture of how the news sites responded to the tragedy, and some solid information on how companies might minimize the impact on their own sites when demand suddenly shoots through the roof.

Keynote Web Statistics for April 15
Keynote's data shows a spike in response time and drop in reliability corresponding to the time of the explosions in Boston.
Keynote's data shows a spike in response time and drop in reliability corresponding to the time of the explosions in Boston.

I spoke with Aaron Rudger, web performance expert (a title, not a description) with Keynote Systems. He told me that keynotes index is built by performing the same query against 22 different websites on a regular schedule throughout the day. "We take measurements using our desktop web performance measurement technology, which on a regular, repeated basis captures a measurement of the individual site's availability and performance," he said. Yesterday, they saw something unusual in their test results.

The test used to create the results in the image come from requesting the site's home page. The download time (in seconds) is shown in blue, while the success rate is shown in orange. Rudger said that there are a number of ways to begin peeling back the layers in order to understand the meaning of the data they receive:

One of the ways is looking at the dynamic, to see whether the response and uptime are consistent for every site we examine. In this case, we have [the test routine] running from 10 different places in the US: San Francisco, New York, Boston, and others are represented. In this case, we didn't see any appreciable difference in results across the agents, so it suggests that this wasn't primarily a network congestion issue within the ISPs in a certain region. We didn't for example, see a degradation in the New England area. We saw the performance degrade across the nation, which indicates that demand is an issue rather than the area.

I asked Rudger whether companies might predict the scale of a potential traffic spike, build to that spike and build an infrastructure to meet the need, or whether some spikes are simply too great for any server to bear. He told me that it was obvious that some sites were able to deal with the demand better than others, but that it was more than a simple matter of building an infrastructure that can scale to meet demand.

Rudger said that Google News, for example, is a very different site than There are dramatic differences in the content provided and the way in which it is presented to the user, and dramatic differences in performance between the sites as well. He said that the performance might well have to do with the infrastructure on which each site is built, but there can also be a huge performance impact that is a function of how the content is provided or rendered by the sites. The site's performance, Rudger said, could be associated with the still images or video that are provided on the page.

One key, he said, is understanding the relationship between your site's performance and the performance of content that might be provided by a third party. "With many sites, being able to provide a good experience to the user means that, if you're depending on content from another provider, you have to deal with that connection as a point of failure," Rudger said. "The third-party provider could be the point of weakness that slows things down."

So what lessons can the CIO of a non-media company take away from the performance hit suffered by CNN, NBC, and the other websites yesterday? Rudger has a few key suggestions:

We definitely preach that, if there's any way that you can reasonably create a fairly predictive high-demand scenario, imagine that a spike in demand is a fairly certain occurrence and test to make sure that the infrastructure is capable of meeting that demand. The other thing is the third-party dependencies: We emphasize that site owners need to understand what the dependencies are, employ mitigation strategies so you can decouple them when performance starts to degrade, and have a mechanism in place to monitor the impact of the third-party dependencies from a performance point of view.

View Comments: Newest First | Oldest First | Threaded View
Page 1 / 5   >   >>
Henrisha   IT Lessons From a Tragedy   5/3/2013 5:39:28 AM
Re: How do you handle spikes?
True. The fact that people don't really fact-check or research the information that they eventually end up posting lets them post updates and snippets faster. However, as we have seen in recent events, this has also given rise to errors and misconceptions. News networks make mistakes as well, but probably with a lesser frequency.
sohaibmasood   IT Lessons From a Tragedy   4/27/2013 1:39:32 AM
Re: tragedy indeed.
Curt, I also follow people from different walks of life for updates on different things. Filtering hash tags has become difficult for me too. At times I don't know which news I should value more. So, what I do is that I only value needs from my trusted circle. 

I ONLY use Twitter on my smartphone. I have never been to their website. I think most people here do it the same way. That is why we consider it to be our primary source of info on the go. 
CurtisFranklin   IT Lessons From a Tragedy   4/23/2013 11:10:23 AM
Re: How do you handle spikes?
@singlemud, you're absolutely correct: the correlation and confirmation service should be a key piece of what the established media offers. The problem for the news organizations comes when, as with several aspects of the story last week, they get it wrong. I think that many people ask a very legitimate question: If a news organization is no more reliable than my Facebook feed, then why do I need the news organization?

As with most forms of competition, social networks are going to force news organizations to improve, or they're going to disappear.
CurtisFranklin   IT Lessons From a Tragedy   4/23/2013 11:07:56 AM
Re: tragedy indeed.
@sohaib, the constant update capability of Twitter is important. It's interesting: I follow a mix of folks, some for entertainment, some for news, some because they're friends. I should probably get a bit more sophisticated in how I filter on hashtags when it comes to major events -- I suppose that, like any new medium, there's a learning curve associated with getting the most from the capabilities!

One thing that I don't do is consume Twitter primarily on my mobile devices. Is it your sense that most people do get their Twitter feeds on their mobile? That would certainly make a huge difference in the value I get from the service...
singlemud   IT Lessons From a Tragedy   4/22/2013 12:51:01 PM
Re: How do you handle spikes?
There is no way for the big media like CNN to compete with facebook friends update. because CNN need to filter or verify the accuracy of the news because put it online, which will take several minutes at least.
impactnow   IT Lessons From a Tragedy   4/22/2013 12:47:34 PM
Re: How do you handle spikes?
Curt I agree the news is no longer owned by the reporters it is now more of a shared endeavor consisting of several news outlets complemented by people in locala reas we are seeing more of this over the past year when accounts were often local people from places like Sandy locations, the Texas explosion etc.
CurtisFranklin   IT Lessons From a Tragedy   4/21/2013 10:44:32 PM
Re: How do you handle spikes?
@freespiritny25, I don't use Facebook as my primary source, but here's something I found interesting: On Friday evening, when the capture of the younger brother was winding toward its conclusion, I was having dinner at a restaurant in Orlando. They had CNN on television screens, and I was using my mobile phone to conduct a Facebook chat with a friend in Boston.

I was getting updates from my friend in Boston 3 - 5 minutes before CNN reported the events. The information I got from my friend was just as accurate as the CNN feed, and slightly faster -- and came complete with commentary on the things that I found important.

I suspect that there were many thousands of people like me on Friday evening: The question for the big news outlets is how to compete, not with one another, but with a viewer's friend who happens to be closer to the event. I'm not sure that any of us have quite worked that out, just yet.
freespiritny25   IT Lessons From a Tragedy   4/21/2013 9:19:53 PM
Re: How do you handle spikes?
I'm not ready to use Facebook as my primary source of news.
MDMConsult   IT Lessons From a Tragedy   4/20/2013 9:33:52 PM
Re: Preparedness is the key
Yes, the shifts are changing too fast. The media does do well today in how new mediums like social media are being used. The way in which we use Twitter has definitely evolved and how users leverage the tool, it is done in various ways. It does bring us closer, more connected quicker through real time information.
sohaibmasood   IT Lessons From a Tragedy   4/19/2013 12:51:54 PM
Re: tragedy indeed.
Curt, it has definitely helped us here. Reports & anchors usually tweet about the current happenings almost instantly. Not to forget the official twitter accounts of news agencies that constantly pump updates. 
Page 1 / 5   >   >>

The blogs and comments posted on do not reflect the views of TechWeb,, or its sponsors., TechWeb, and its sponsors do not assume responsibility for any comments, claims, or opinions made by authors and bloggers. They are no substitute for your own research and should not be relied upon for trading or any other purpose.

More Blogs from Curtis Franklin Jr.
Curtis Franklin Jr.   5/30/2014   10 comments
A good community can teach you a lot. And Enterprise Efficiency has been one of the best.
Curtis Franklin Jr.   5/26/2014   41 comments
Today is Memorial Day in the US, a day for remembering those who gave, in the words of Abraham Lincoln, "the last full measure of devotion" for their country and its citizens. It is a ...
Curtis Franklin Jr.   5/22/2014   35 comments
You're about to know precisely where your customers are and what they're doing. Are you ready for Big Data Advertising Everywhere?
Curtis Franklin Jr.   5/6/2014   19 comments
PHP is a great tool for building web pages that access databases. It's pretty nifty for pwning an enterprise site, too.
Curtis Franklin Jr.   4/30/2014   36 comments
BASIC turns 50 this year. Many IT pros wrote their first line of code in the venerable language, but is the ability to write code even important at the top of the IT ladder?
Latest Archived Broadcast
We talk with Bernard Golden about accelerating application delivery in the cloud.
On-demand Video with Chat
Register for this video discussion to learn how tablets can provide true business usability and productivity.
E2 IT Migration Zones
IT Migration Zone - UK
Why PowerShell Is Important
Reduce the Windows 8 Footprint for VDI
Rethinking Storage Management
IT Migration Zone - FR
SQL Server : 240 To de mémoire flash pour votre data warehouse
Quand Office vient booster les revenus Cloud et Android de Microsoft
Windows Phone : Nokia veut davantage d'applications (et les utilisateurs aussi)
IT Migration Zone - DE
Cloud Computing: Warum Unternehmen trotz NSA auf die „private“ Wolke setzen sollten
Cloud Computing bleibt Wachstumsmarkt – Windows Azure ist Vorreiter
Like Us on Facebook
Twitter Feed
Enterprise Efficiency Twitter Feed
Site Moderators Wanted
Enterprise Efficiency is looking for engaged readers to moderate the message boards on this site. Engage in high-IQ conversations with IT industry leaders; earn kudos and perks. Interested? E-mail:
Dell's Efficiency Modeling Tool
The major problem facing the CIO is how to measure the effectiveness of the IT department. Learn how Dell’s Efficiency Modeling Tool gives the CIO two clear, powerful numbers: Efficiency Quotient and Impact Quotient. These numbers can be transforma¬tive not only to the department, but to the entire enterprise.

Read the full report
The State of Enterprise Efficiency in the Virtual Era: Virtualization – Smart Approaches to Maximize Gains
Virtualization is a presence in nearly all enterprise data centers. But not all companies are using it to its best effect. Learn the common characteristics of success, what barriers companies face, and how to get the most from your efforts.

Read the full report
Informed CIO: Dollars & Sense: Virtual Desktop Infrastructure
Cut through the VDI hype and get the full picture -- including ROI and the impact on your Data Center -- to make an informed decision about your virtual desktop infrastructure deployments.

Read the full report
A Video Case Study – Translational Genomics Research Institute
e2 Video

On the Case
TGen IT: Where We're Going Next

7|11|12   |   08:12   |   10 comments

Now that TGen has broken new ground in genomic research by using Dell's storage, cloud, and high-performance computing solutions, the company discusses what will come next for it and for personalized medicine.
On the Case
Better Care Through Better Communications

6|6|12   |   02:24   |   11 comments

The achievements of the TGen/Dell project could improve how all people receive healthcare, because they are creating ways to improve end-to-end communication of medical data.
On the Case
TGen IT: Where We Are Now

5|15|12   |   06:58   |   6 comments

TGen is breaking new ground in genomic research by using Dell's storage, cloud, and high-performance computing solutions.
On the Case
TGen IT: Where We Were

4|27|12   |   06:45   |   10 comments

The Translational Genomics Research Institute wanted to save lives, but its efforts were hobbled by immense computing challenges related to collecting, processing, sharing, and storing enormous amounts of data.
On the Case
1,200% Faster

4|18|12   |   02:27   |   12 comments

Through their partnership, Dell and TGen have increased the speed of TGen’s medical research by 1,200 percent.
On the Case
IT May Improve Children's Chances of Survival

4|17|12   |   02:12   |   8 comments

IT is helping medical researchers reach breakthroughs in a way and pace never seen before.
On the Case
Medical Advances in the Cloud

4|10|12   |   1:25   |   5 comments

TGen and Dell are pushing the boundaries of computing, and harnessing the power of the cloud to improve healthcare.
On the Case
TGen: Living the Mission

4|9|12   |   2:25   |   3 comments

TGen's CIO puts the organizational mission at the heart of everything the IT staff does.
On the Case
TGen Speeding Up Biomedical Research to Save More Lives

4|5|12   |   1:59   |   6 comments

The Translational Genomics Research Institute is revamping its computing to improve speed, storage, and collaboration – and, most importantly, to save lives.
On the Case
Computing Power Helping to Save Children's Lives

3|28|12   |   2:13   |   3 comments

The Translational Genomics Institute’s partnership with Dell is enabling them to treat kids with neuroblastoma more quickly and save more lives.
Tom Nolle
The Big Reason to Use Office

3|18|14   |   02:24   |   46 comments

Office and personal productivity tools come in a first-class and coach flavor set, but what makes the difference is primarily little things that most users won't encounter. What's the big issue in using something other than Office, and can you get around it?
E2 Editors
SPONSORED: Mobile Security — A Use Case

3|4|14   |   04:27   |   16 comments

New mobile security solutions can accommodate a wide array of needs, including those of a complex university environment.
Tom Nolle
Killing Net Neutrality Might Save You Money

1|16|14   |   2:13   |   16 comments

The DC Court of Appeals voided most of the Neutrality Order, and whatever it might mean for the Internet overall, it might mean better and cheaper Internet VPNs for businesses.
Tom Nolle
The Internet of Everythinguseful

1|10|14   |   2:18   |   19 comments

We really don't want an "Internet of Everything" but even building an Internet of Everythinguseful means setting some ground rules to insure there's value in the process and that costs and risks are minimized.
Tom Nolle
Maturing Google Chrome

12|30|13   |   2.18   |   25 comments

Google's Chrome OS has a lot of potential value and a lot of recent press, but it still needs something to make it more than a thin client. It needs cloud integration, it needs extended APIs via web services, and it needs to suck it up and support a hard drive.
Sara Peters
No More Cookie-Cutter IT

12|23|13   |   03.58   |   21 comments

Creating the right combination of technology, people, and processes for your IT organization is a lot like baking Christmas cookies.
Sara Peters
Smart Wigs Not a Smart Idea

12|5|13   |   3:01   |   46 comments

Sony is seeking a patent for wigs that contain computing devices.
Tom Nolle
Cloud in the Wild

12|4|13   |   02:23   |   15 comments

On a recent African trip I saw examples of the value of the cloud in developing nations, for educational and community development programs. We could build on this, but not only in developing economies, because these same programs are often under-supported even in first-world countries.
E2 Editors
SPONSORED: Is Malware Evading Your IPS?

11|18|13   |   03:16   |   4 comments

Intrusion prevention software is supposed to detect and block malware intrusions, but clever malware authors can evade your IPS in these five main ways.
Sara Peters
Where Have All the Mentors Gone?

9|27|13   |   3:15   |   38 comments

A good professional mentor can change your life for the better... but where do you find one?
Tom Nolle
SDN Wars & You Could Win

9|17|13   |   2:10   |   5 comments

VMware's debate with Cisco on SDN might finally create a fusion between an SDN view that's all about software and another that's all about network equipment. That would be good for every enterprise considering the cloud and SDN.
Ivan Schneider
The Future of the Smart Watch

9|12|13   |   3:19   |   39 comments

Wearing a bulky, oversized watch is good training for the next phase in wristwatches: the Internet-enabled, connected watch. Why the smartphone-tethered connected watch makes sense, plus Ivan demos an entirely new concept for the "smart watch."
Tom Nolle
Cutting Your Cloud Storage Costs

9|4|13   |   2:06   |   3 comments

Cloud storage costs are determined primarily by the rate at which files are changed and the possibility of concurrent access/update. If you can structure your storage use to optimize these factors you can cut costs, perhaps to zero.
Sara Peters
Do CIOs Need an IT Background?

8|29|13   |   2:11   |   23 comments

Most of the CIOs interviewed in the How to Become a CIO series did not start their careers as IT professionals. So is an IT background essential?
Ivan Schneider
The Internet Loves Birthdays

8|27|13   |   3:25   |   69 comments

The Internet has evolved into a machine for drumming up a chorus of "Happy Birthday" messages, from family, friends, friends of friends who you added on Facebook, random people that you circled on G+, and increasingly, automated bots. Enough already.