Tech Errors Are Part of Stock Trading Risk

Sara Peters, Editor in Chief | 1/17/2013 | 25 comments

Sara Peters
All trading on the stock market comes with risk. The potential for technological errors and disruptions in stock exchanges is now just a standard variable in the risk equation. Traders have to account for it when they're deciding which exchanges to use. And the exchanges have to account for it when they're deciding how to strike the right balance between the need for speed and the potential for error.

In the past few weeks, the U.S. stock exchanges have been plagued with technological snafus -- including multiple service disruptions (one that lasted nearly an hour), data reporting system malfunctions, and the discovery of a programming mistake that, over the past four years, caused a total of roughly 435,000 trades to be executed at the wrong price. Last week, the New York Times documented the recent string of mishaps. From the story:

Regulators and traders have said that malfunctions are inevitable in any complex computer system. But many of these same people say that such problems were less frequent before the nation’s stock exchanges were thrown into a technological arms race in the middle of the last decade as a host of upstart exchanges like BATS [Global Markets] challenged incumbents like the New York Stock Exchange.

Is this "technological arms race" a real thing, and if so, what are CIOs in stock exchanges, and the financial services industry at large, supposed to do to minimize the negative impact this arms race is having on trading software quality assurance?

"It's real," says Steve Rubinow, CIO of FX Alliance and former CIO of NYSE Euronext, "because for years now, the customers have said that they can have an advantage if they can be a fraction of a second in front of someone else."

To stay competitive, stock exchanges have pushed the limits of technology to shave down the time it takes to complete a transaction to mere microseconds. Therefore, any software error or service disruption can make a big impact very quickly.

"Considering the fact that these systems have millions of transactions a second, with individual component response times of microseconds," said Rubinow, "while I'm not saying that problems are ever acceptable, I would say that the track record overall isn't bad, given the proper context."

Would investors agree? Or does every little technological slip-up chip away at investors' confidence, and ultimately have a damaging impact on the whole market?

"As a general principle I would say no," said Rubinow. “When things go awry the effects can be profound. However, one has to assume that there's going to be no perfect systems. That they will fail from time to time or just act oddly, so you must be prepared to properly address most reasonable eventualities."

Fair enough. Risk management is a trader's bread and butter, after all, so they should be able to handle the possibility of things going awry on the software front. Nevertheless, if you're the CIO of a stock exchange, you'd prefer it if things went awry with somebody else's software -- preferably one of your competitor's. The best way to avoid system errors, of course, is to do a stellar job at quality assurance and lots and lots of testing. But that's easier said than done. Because the stock market is so complex and intertwined, thorough testing means you have to consider not just your own internal systems and situations, but also the systems and situations of all the other players in the marketplace.

"It is impossible to test for all possible conditions," said Rubinow. "And you can’t take forever, because you’re trying to be competitive."

Rubinow said that trading software could be made more pristine if everyone conducted more elaborate testing over longer periods of time and collaborated with the other players in the market ecosystem. But, "it would take more time and involve more people, it would be more costly, and still we couldn't simulate everything, so it is unlikely that the system would be bulletproof."

Well, there goes that idea. If that degree of elaborate testing isn't possible, at least make sure that your developers get the software as close to perfect as possible before it's rolled out... and put in controls that let you know when it proves itself to be imperfect.

Despite business pressures for speed, Rubinow warns CIOs and development teams against falling into the trap of pushing out inadequately tested code just because they're pressed for time. "Code has to be robust when it leaves a developer's hands and goes to the quality assurance department,” he said. “Quality and testing is everyone’s responsibility.”

"Organizations need to have risk mitigation tools and controls that kick in," said Rubinow. "It's important to be able to recognize when something is out of the ordinary and then the system should first signal it, or if it's so egregious then it should be shut off what's going on is understood the market can return to normal."

Another option, of course, is to get scared and steer clear of all these technological advancements that make trading faster. But Rubinow would advise against being too risk-averse.

"Those who are swift of foot and can run with the changes, they have all kinds of opportunities," said Rubinow. "If you stop to ponder for a few years you're going to be left in the dust."

And why get left in the dust when there are truly good reasons to keep up with the changes?

"Implying that it's an arms race minimizes the fact that technology advances and the speed has had advantages, for example delivering efficiency for investors," said Rubinow. And he doesn't see the push towards greater speed lessening. "As long as there is a financial incentive, people are going to continue to chase it."

View Comments: Newest First | Oldest First | Threaded View
Page 1 / 3   >   >>
Technocrat   Tech Errors Are Part of Stock Trading Risk   1/24/2013 1:29:29 PM
Pressure and The Role of IT
Really interesting to ponder the pressure that Finance based tech professionals are under.  I know how much pressure is applied when things don't go right in my non-financial based tech position.  Seems like a nice challenge, though I have heard that IT is not for the most part appreciated for their efforts in this industry.  Amazing that there are many who still think they can do things ( be successful) without IT
Sara Peters   Tech Errors Are Part of Stock Trading Risk   1/23/2013 8:21:44 AM
Re: accept the bug in the machine
Trek. Ahhh! Well that's a fascinating idea: "what I have in mindnis a speed limit for transactions. We consider doping cheating in sports, so isn't technology an unfair advantage as well?" this way of equating this to steroid use is tough, because i see both similarities and difference. Since sporting events are intended to show the pinnacle of human achievement. (without artificial enhancements) and the fact that some people use these substances to give them an unfair advantage. Steroids negatively impact the sport. However i think that most of the trading tech is available to all exchanges. However the risk of highest-speed trading puts the investors (the specators) at risk, not just the market itself (the sport). Hmmm...
sohaibmasood   Tech Errors Are Part of Stock Trading Risk   1/23/2013 6:57:40 AM
Re: NY Stock Exchange getting bought...
Certainly seems like a very big step for a technology oriented company. A trading company without a floor is all set to buy one of the biggest trading floors in the world. Interesting times! 
Trek   Tech Errors Are Part of Stock Trading Risk   1/22/2013 4:48:46 PM
Re: accept the bug in the machine
@ Sara -  In an idea world, any person should be compensating for any trading errors, though this could possibly include millions of traders.     What I have in mind is a speed limit for transactions. We consider doping cheating in sports, so isn't technology an unfair advantage as well.  

A Speed Limit for the Stock Market - The New York Times

The thing is these technical errors are common.Here are three that happened between Oct and Aug 2012 alson. 

Error by Knight Capital rips through stock market | Reuters

Glitch wipes nearly $60bn off Indian stock index | Business ...

Exchanges hit again by trading error -


Sara Peters   Tech Errors Are Part of Stock Trading Risk   1/22/2013 3:16:59 PM
Re: accept the bug in the machine
More intriguing stuff from the NYT story about ICE buying the New York Stock Exchange. It relates to this conversation we've been having about how quickly the market can be affected in these days of high-speed trading, and how a stock exchange itself might put limits on its own speed, even if other regulators won't.

One sign of trouble came in 2010, when an errant trade ricocheted through computer networks and touched off one of the most harrowing moments in stock market history. The Dow Jones industrial average plunged 900 points in a matter of minutes, and a new phrase entered the lexicon: flash crash.

Since then, flash crashes in individual stocks have been remarkably common, as the centuries-old system of central exchanges has given way to a field of competing electronic systems.

ICE wasn't involved in any of these problems. In fact, it has been praised as one of the first exchanges to put limits on lightning-quick, high-frequency trading. This points to Mr. Sprecher's deftness in piloting his company through periods of regulation, deregulation and now re-regulation.
Sara Peters   Tech Errors Are Part of Stock Trading Risk   1/22/2013 2:37:33 PM
NY Stock Exchange getting bought...
In case you didn't know, the New York Stock Exchange is on its way to being bought by IntercontinentalExchange. And as the NY Times explains it "How the New York Stock Exchange fell into Mr. Sprecher's hands is, at heart, a story of the disruptive power of innovation. ICE, as IntercontinentalExchange is known, did not even exist 13 years ago. It has no cavernous trading floor, no gilded halls, no sweaty brokers braying for money on the financial markets. What it has is technology."
Cyrus   Tech Errors Are Part of Stock Trading Risk   1/22/2013 11:47:53 AM
Re: Tech errors does not mean to be losses
@singlemud That's not correct because in essence your customer (be it a mutual fund, pension fund, money manager, etc) expects someone who mismanages a legitimate trade to make good on issues relating to pricing and overall transaction processessing that are in their control.

So in effect when you have software and other systems/processes that aren't ready for prime time and/or that hiccup in business-critical environments, someone will always pay.
Sara Peters   Tech Errors Are Part of Stock Trading Risk   1/22/2013 11:25:52 AM
Re: accept the bug in the machine
@Trek  "My feelings on it is that errors should have a zero tolerance policy." How do you think a "zero tolerance policy" should be enforced? Should the stock exchanges which develop/use flawed software be fined? Shut down?
Sara Peters   Tech Errors Are Part of Stock Trading Risk   1/22/2013 11:20:22 AM
Re: accept the bug in the machine
@SunitaT Well, here is the one of the main challenges: "introduction of automated algorithm trading has made things worse because it cumulates the error." Without automation the stock exchanges would never be able to handle so many transactions -- humans alone simply couldn't keep up with the pace. And even if they could, there would still be errors, because humans aren't perfect. After all, it's humans who create imperfect code.
SunitaT   Tech Errors Are Part of Stock Trading Risk   1/19/2013 6:44:53 AM
Re: accept the bug in the machine
And when you have millions of transactions per second happening, how easy is it to locate an error or even be sure one occurred.

@Trek, I think it would be very difficult to locate an error because millions of transactions per second. Moreoever introduction of automated algorithm trading has made things worse because it cumulates the error.
Page 1 / 3   >   >>

The blogs and comments posted on do not reflect the views of TechWeb,, or its sponsors., TechWeb, and its sponsors do not assume responsibility for any comments, claims, or opinions made by authors and bloggers. They are no substitute for your own research and should not be relied upon for trading or any other purpose.

More Blogs from Sara Peters
Sara Peters   5/30/2014   18 comments
Before I throw my knapsack over my shoulder, head out the door, slouch my way down lonely roads, and try to get a kind stranger to give me a lift to DarkReading, I want to bid a fond adieu ...
Sara Peters   5/7/2014   55 comments
There's a new player in the wearable technology market. While it has a new attitude and a new business plan, it also seems to have new problems that may cause it to miss the mark and the ...
Sara Peters   5/1/2014   32 comments
It's time for an intervention. This addiction to Windows XP has gone too far.
Sara Peters   4/10/2014   21 comments
The race is on to restore trust in the very security tools that are used to ensure online trust. Websites, applications, and services are hastening to patch Heartbleed, a flaw in the ...
Sara Peters   4/4/2014   17 comments
Tablets are becoming more affordable all the time, but they're not so inexpensive that you wouldn't worry if kindergartners treated them with the same care they treat construction paper ...
Latest Archived Broadcast
We talk with Bernard Golden about accelerating application delivery in the cloud.
On-demand Video with Chat
Register for this video discussion to learn how tablets can provide true business usability and productivity.
E2 IT Migration Zones
IT Migration Zone - UK
Why PowerShell Is Important
Reduce the Windows 8 Footprint for VDI
Rethinking Storage Management
IT Migration Zone - FR
SQL Server : 240 To de mémoire flash pour votre data warehouse
Quand Office vient booster les revenus Cloud et Android de Microsoft
Windows Phone : Nokia veut davantage d'applications (et les utilisateurs aussi)
IT Migration Zone - DE
Cloud Computing: Warum Unternehmen trotz NSA auf die „private“ Wolke setzen sollten
Cloud Computing bleibt Wachstumsmarkt – Windows Azure ist Vorreiter
Like Us on Facebook
Twitter Feed
Enterprise Efficiency Twitter Feed
Site Moderators Wanted
Enterprise Efficiency is looking for engaged readers to moderate the message boards on this site. Engage in high-IQ conversations with IT industry leaders; earn kudos and perks. Interested? E-mail:
[email protected]
Dell's Efficiency Modeling Tool
The major problem facing the CIO is how to measure the effectiveness of the IT department. Learn how Dell’s Efficiency Modeling Tool gives the CIO two clear, powerful numbers: Efficiency Quotient and Impact Quotient. These numbers can be transforma¬tive not only to the department, but to the entire enterprise.

Read the full report
The State of Enterprise Efficiency in the Virtual Era: Virtualization – Smart Approaches to Maximize Gains
Virtualization is a presence in nearly all enterprise data centers. But not all companies are using it to its best effect. Learn the common characteristics of success, what barriers companies face, and how to get the most from your efforts.

Read the full report
Informed CIO: Dollars & Sense: Virtual Desktop Infrastructure
Cut through the VDI hype and get the full picture -- including ROI and the impact on your Data Center -- to make an informed decision about your virtual desktop infrastructure deployments.

Read the full report
A Video Case Study – Translational Genomics Research Institute
e2 Video

On the Case
TGen IT: Where We're Going Next

7|11|12   |   08:12   |   10 comments

Now that TGen has broken new ground in genomic research by using Dell's storage, cloud, and high-performance computing solutions, the company discusses what will come next for it and for personalized medicine.
On the Case
Better Care Through Better Communications

6|6|12   |   02:24   |   11 comments

The achievements of the TGen/Dell project could improve how all people receive healthcare, because they are creating ways to improve end-to-end communication of medical data.
On the Case
TGen IT: Where We Are Now

5|15|12   |   06:58   |   6 comments

TGen is breaking new ground in genomic research by using Dell's storage, cloud, and high-performance computing solutions.
On the Case
TGen IT: Where We Were

4|27|12   |   06:45   |   10 comments

The Translational Genomics Research Institute wanted to save lives, but its efforts were hobbled by immense computing challenges related to collecting, processing, sharing, and storing enormous amounts of data.
On the Case
1,200% Faster

4|18|12   |   02:27   |   12 comments

Through their partnership, Dell and TGen have increased the speed of TGen’s medical research by 1,200 percent.
On the Case
IT May Improve Children's Chances of Survival

4|17|12   |   02:12   |   8 comments

IT is helping medical researchers reach breakthroughs in a way and pace never seen before.
On the Case
Medical Advances in the Cloud

4|10|12   |   1:25   |   5 comments

TGen and Dell are pushing the boundaries of computing, and harnessing the power of the cloud to improve healthcare.
On the Case
TGen: Living the Mission

4|9|12   |   2:25   |   3 comments

TGen's CIO puts the organizational mission at the heart of everything the IT staff does.
On the Case
TGen Speeding Up Biomedical Research to Save More Lives

4|5|12   |   1:59   |   6 comments

The Translational Genomics Research Institute is revamping its computing to improve speed, storage, and collaboration – and, most importantly, to save lives.
On the Case
Computing Power Helping to Save Children's Lives

3|28|12   |   2:13   |   3 comments

The Translational Genomics Institute’s partnership with Dell is enabling them to treat kids with neuroblastoma more quickly and save more lives.
Tom Nolle
The Big Reason to Use Office

3|18|14   |   02:24   |   46 comments

Office and personal productivity tools come in a first-class and coach flavor set, but what makes the difference is primarily little things that most users won't encounter. What's the big issue in using something other than Office, and can you get around it?
E2 Editors
SPONSORED: Mobile Security — A Use Case

3|4|14   |   04:27   |   16 comments

New mobile security solutions can accommodate a wide array of needs, including those of a complex university environment.
Tom Nolle
Killing Net Neutrality Might Save You Money

1|16|14   |   2:13   |   16 comments

The DC Court of Appeals voided most of the Neutrality Order, and whatever it might mean for the Internet overall, it might mean better and cheaper Internet VPNs for businesses.
Tom Nolle
The Internet of Everythinguseful

1|10|14   |   2:18   |   19 comments

We really don't want an "Internet of Everything" but even building an Internet of Everythinguseful means setting some ground rules to insure there's value in the process and that costs and risks are minimized.
Tom Nolle
Maturing Google Chrome

12|30|13   |   2.18   |   25 comments

Google's Chrome OS has a lot of potential value and a lot of recent press, but it still needs something to make it more than a thin client. It needs cloud integration, it needs extended APIs via web services, and it needs to suck it up and support a hard drive.
Sara Peters
No More Cookie-Cutter IT

12|23|13   |   03.58   |   21 comments

Creating the right combination of technology, people, and processes for your IT organization is a lot like baking Christmas cookies.
Sara Peters
Smart Wigs Not a Smart Idea

12|5|13   |   3:01   |   46 comments

Sony is seeking a patent for wigs that contain computing devices.
Tom Nolle
Cloud in the Wild

12|4|13   |   02:23   |   15 comments

On a recent African trip I saw examples of the value of the cloud in developing nations, for educational and community development programs. We could build on this, but not only in developing economies, because these same programs are often under-supported even in first-world countries.
E2 Editors
SPONSORED: Is Malware Evading Your IPS?

11|18|13   |   03:16   |   4 comments

Intrusion prevention software is supposed to detect and block malware intrusions, but clever malware authors can evade your IPS in these five main ways.
Sara Peters
Where Have All the Mentors Gone?

9|27|13   |   3:15   |   38 comments

A good professional mentor can change your life for the better... but where do you find one?
Tom Nolle
SDN Wars & You Could Win

9|17|13   |   2:10   |   5 comments

VMware's debate with Cisco on SDN might finally create a fusion between an SDN view that's all about software and another that's all about network equipment. That would be good for every enterprise considering the cloud and SDN.
Ivan Schneider
The Future of the Smart Watch

9|12|13   |   3:19   |   39 comments

Wearing a bulky, oversized watch is good training for the next phase in wristwatches: the Internet-enabled, connected watch. Why the smartphone-tethered connected watch makes sense, plus Ivan demos an entirely new concept for the "smart watch."
Tom Nolle
Cutting Your Cloud Storage Costs

9|4|13   |   2:06   |   3 comments

Cloud storage costs are determined primarily by the rate at which files are changed and the possibility of concurrent access/update. If you can structure your storage use to optimize these factors you can cut costs, perhaps to zero.
Sara Peters
Do CIOs Need an IT Background?

8|29|13   |   2:11   |   23 comments

Most of the CIOs interviewed in the How to Become a CIO series did not start their careers as IT professionals. So is an IT background essential?
Ivan Schneider
The Internet Loves Birthdays

8|27|13   |   3:25   |   69 comments

The Internet has evolved into a machine for drumming up a chorus of "Happy Birthday" messages, from family, friends, friends of friends who you added on Facebook, random people that you circled on G+, and increasingly, automated bots. Enough already.