With news that Google slashed the price of their big-data offering "Big Query" by up to 85 percent, one has to wonder if the move is to ward off competitors -- or simply that the anticipated market for cloud-stored big data isn't turning out to be as massive as once expected. And while cloud-based big data sounds good in theory, many are coming to the realization that the cloud may not be the optimal solution.
Those of us involved in enterprise IT have been bombarded with stories regarding the wonders of big data and how it's being used to revolutionize virtually all aspects of business. So it may come as a surprise that cloud-based big-data usage isn't growing as quickly as originally predicted. In fact, 2014 is now being forecasted to grow at a slower pace than it had in 2013. Why?
For one, there are major barriers to entry that must first be overcome in order to get any big-data project off the ground. Because big data requires a big database, you need to be able to find the technical resources that understand how to operate complex platforms. Many IT departments simply don't have and can't recruit the in-house expertise required to deploy and manage these types of architectures. That's why so many have turned to big-data cloud services as a logical alternative. But even there, many are finding cloud-stored big data to be a poor fit for what they want to do.
If a company already maintains and manages a sizable legacy database with which to start their big-data aspirations, they're stuck with the challenge of handing it over to a service provider to normalize and implement into the new big-data cloud infrastructure. Depending on how well managed the legacy database is, getting it to fit into a new solution could be a major headache.
WAN and Internet costs/bottlenecks should be another top concern when looking to a big-data cloud solution. Many are coming to the realization that the amount of bandwidth needed to operate a cloud big-data architecture far exceeds its usefulness. So before you even consider moving your data into the cloud, make sure you accurately calculate bandwidth usage and management requirements now and several years into the future.
Lastly, as with all cloud ventures, recognize and plan an exit strategy to get your big-data architecture out of any cloud service provider. While data may last forever, cloud providers and partnerships may not. Cloud providers make it fairly easy to extract your database. The difficult part resides in moving all the infrastructure policies and workflows that creates your own, custom cloud architecture. These are things like security access rules, quality of service policies, and specialized application and database flows that are defined within the cloud infrastructure itself. So keep in mind that it's not just the data that you want to be able to easily move, but the also the policies that define cloud parameters.
I'm not seeking to deter anyone from his or her cloud-based big-data aspirations. Rather, I simply want to point out less obvious pitfalls that may not immediately come to mind. And at the end of the day, if you've thoroughly vetted these and other potential risks and still feel it's worth it, good luck with your big-data adventure in the cloud!