Big Data in the Cloud: It's Not for Everyone

Andrew Froehlich, Network Engineer & IT Consultant | 4/16/2014 | 22 comments

Andrew Froehlich
With news that Google slashed the price of their big-data offering "Big Query" by up to 85 percent, one has to wonder if the move is to ward off competitors -- or simply that the anticipated market for cloud-stored big data isn't turning out to be as massive as once expected. And while cloud-based big data sounds good in theory, many are coming to the realization that the cloud may not be the optimal solution.

Those of us involved in enterprise IT have been bombarded with stories regarding the wonders of big data and how it's being used to revolutionize virtually all aspects of business. So it may come as a surprise that cloud-based big-data usage isn't growing as quickly as originally predicted. In fact, 2014 is now being forecasted to grow at a slower pace than it had in 2013. Why?

For one, there are major barriers to entry that must first be overcome in order to get any big-data project off the ground. Because big data requires a big database, you need to be able to find the technical resources that understand how to operate complex platforms. Many IT departments simply don't have and can't recruit the in-house expertise required to deploy and manage these types of architectures. That's why so many have turned to big-data cloud services as a logical alternative. But even there, many are finding cloud-stored big data to be a poor fit for what they want to do.

If a company already maintains and manages a sizable legacy database with which to start their big-data aspirations, they're stuck with the challenge of handing it over to a service provider to normalize and implement into the new big-data cloud infrastructure. Depending on how well managed the legacy database is, getting it to fit into a new solution could be a major headache.

WAN and Internet costs/bottlenecks should be another top concern when looking to a big-data cloud solution. Many are coming to the realization that the amount of bandwidth needed to operate a cloud big-data architecture far exceeds its usefulness. So before you even consider moving your data into the cloud, make sure you accurately calculate bandwidth usage and management requirements now and several years into the future.

Lastly, as with all cloud ventures, recognize and plan an exit strategy to get your big-data architecture out of any cloud service provider. While data may last forever, cloud providers and partnerships may not. Cloud providers make it fairly easy to extract your database. The difficult part resides in moving all the infrastructure policies and workflows that creates your own, custom cloud architecture. These are things like security access rules, quality of service policies, and specialized application and database flows that are defined within the cloud infrastructure itself. So keep in mind that it's not just the data that you want to be able to easily move, but the also the policies that define cloud parameters.

I'm not seeking to deter anyone from his or her cloud-based big-data aspirations. Rather, I simply want to point out less obvious pitfalls that may not immediately come to mind. And at the end of the day, if you've thoroughly vetted these and other potential risks and still feel it's worth it, good luck with your big-data adventure in the cloud!

View Comments: Newest First | Oldest First | Threaded View
Page 1 / 3   >   >>
kstaron   Big Data in the Cloud: It's Not for Everyone   4/29/2014 11:19:38 AM
Another expense
Another obstacle is how you plan on using big data. If you don't have big plans to offset the expense of taking that dats off site and crunching it into something useable it just another expense even if the price was slashed 85%.
singlemud   Big Data in the Cloud: It's Not for Everyone   4/28/2014 2:59:05 PM
Re: Big Data in the Cloud
very good article, I guess that is why the big companies like google are offering big data services. It is very expensive for in-house big data apps
WaqasAltaf   Big Data in the Cloud: It's Not for Everyone   4/24/2014 12:11:59 PM
Re: Build it and they may come...
Andrew, yes this statement will change. Realization of the fact that knowledge of the factors that affect business performance is the key to success and that big data can be managed more effectively and cost-efficiently on cloud will trigger this change.
WaqasAltaf   Big Data in the Cloud: It's Not for Everyone   4/24/2014 12:07:20 PM
Re: Build it and they may come...
MDMConsult, true. This data management art comes from experience, knowledge of variables that affect business performance and understanding of how data is created. Companies now expect every individual to have a data scientist inculcated within them.
tekedge   Big Data in the Cloud: It's Not for Everyone   4/23/2014 7:58:28 PM
Big Data in the Cloud
I have thought about these points at different times when faced with different scenarios but have never been able to put them together this well. Thanks for doing it so well. This has helped me organise my thoughts well so that can give good input when called upon my opinion on this subject....
WaqasAltaf   Big Data in the Cloud: It's Not for Everyone   4/22/2014 1:20:38 PM
Re: Build it and they may come...
Henrisha, true. The IT department plays a vital consultancy role. It must understand which applications and systems are suited for which purpose. If it doesn't, the business user can go wrong in a decision in a way that may even be devastating.
Henrisha   Big Data in the Cloud: It's Not for Everyone   4/22/2014 4:07:44 AM
Re: Build it and they may come...
The goal is to come up with products and services tailored to the target markets, from SMBs to big corporations. A lot of people are actually getting with the cloud and trying it out--it doesn't work for all, but sometimes that's because they're opting for a solution that doesn't work for their purpose.
MDMConsult   Big Data in the Cloud: It's Not for Everyone   4/21/2014 8:29:42 AM
Re: Build it and they may come...
@Hospice_Houngbo Consumers have the ability to research offers online and other options. The current economic climate has resulted in increased pressure for these organization to retain loyal customers. Securing greater customer loyalty can be substantial.
MDMConsult   Big Data in the Cloud: It's Not for Everyone   4/21/2014 8:27:12 AM
Re: Build it and they may come...
Embracing emerging technologies like cloud computing and big data, enable the company to  sustain its competitive advantages and chase down big players.   
WaqasAltaf   Big Data in the Cloud: It's Not for Everyone   4/20/2014 11:03:03 AM
Re: Build it and they may come...
Pedro, do you think that companies get swayed away by too much promotion of cloud-based big data storages on forums and social media or they are making choices that make sense ? 
Page 1 / 3   >   >>

The blogs and comments posted on do not reflect the views of TechWeb,, or its sponsors., TechWeb, and its sponsors do not assume responsibility for any comments, claims, or opinions made by authors and bloggers. They are no substitute for your own research and should not be relied upon for trading or any other purpose.

More Blogs from Andrew Froehlich
Andrew Froehlich   5/6/2014   13 comments
Not all clouds are equal. That's a pretty obvious statement that we can all agree on. Cloud service providers offer differing levels of services, redundancy, and customer service -- all at ...
Andrew Froehlich   4/30/2014   10 comments
In order for enterprise employees to work together as one unified group, they must follow carefully written policies and procedures -- but every once in a while, you may find yourself in ...
Andrew Froehlich   4/22/2014   49 comments
For those of us who study enterprise IT security, last year's Target store hack turned out to be a fantastic case study that was loaded with lessons to learn.
Andrew Froehlich   3/18/2014   27 comments
At a recent South By Southwest Interactive conference in Austin, Edward Snowden said the NSA is "setting fire to the future of the Internet." In light of this, the World Wide Web ...
Latest Archived Broadcast
We talk with Bernard Golden about accelerating application delivery in the cloud.
On-demand Video with Chat
Register for this video discussion to learn how tablets can provide true business usability and productivity.
E2 IT Migration Zones
IT Migration Zone - UK
Why PowerShell Is Important
Reduce the Windows 8 Footprint for VDI
Rethinking Storage Management
IT Migration Zone - FR
SQL Server : 240 To de mémoire flash pour votre data warehouse
Quand Office vient booster les revenus Cloud et Android de Microsoft
Windows Phone : Nokia veut davantage d'applications (et les utilisateurs aussi)
IT Migration Zone - DE
Cloud Computing: Warum Unternehmen trotz NSA auf die „private“ Wolke setzen sollten
Cloud Computing bleibt Wachstumsmarkt – Windows Azure ist Vorreiter
Like Us on Facebook
Twitter Feed
Enterprise Efficiency Twitter Feed
Site Moderators Wanted
Enterprise Efficiency is looking for engaged readers to moderate the message boards on this site. Engage in high-IQ conversations with IT industry leaders; earn kudos and perks. Interested? E-mail:
[email protected]
Dell's Efficiency Modeling Tool
The major problem facing the CIO is how to measure the effectiveness of the IT department. Learn how Dell’s Efficiency Modeling Tool gives the CIO two clear, powerful numbers: Efficiency Quotient and Impact Quotient. These numbers can be transforma¬tive not only to the department, but to the entire enterprise.

Read the full report
The State of Enterprise Efficiency in the Virtual Era: Virtualization – Smart Approaches to Maximize Gains
Virtualization is a presence in nearly all enterprise data centers. But not all companies are using it to its best effect. Learn the common characteristics of success, what barriers companies face, and how to get the most from your efforts.

Read the full report
Informed CIO: Dollars & Sense: Virtual Desktop Infrastructure
Cut through the VDI hype and get the full picture -- including ROI and the impact on your Data Center -- to make an informed decision about your virtual desktop infrastructure deployments.

Read the full report
A Video Case Study – Translational Genomics Research Institute
e2 Video

On the Case
TGen IT: Where We're Going Next

7|11|12   |   08:12   |   10 comments

Now that TGen has broken new ground in genomic research by using Dell's storage, cloud, and high-performance computing solutions, the company discusses what will come next for it and for personalized medicine.
On the Case
Better Care Through Better Communications

6|6|12   |   02:24   |   11 comments

The achievements of the TGen/Dell project could improve how all people receive healthcare, because they are creating ways to improve end-to-end communication of medical data.
On the Case
TGen IT: Where We Are Now

5|15|12   |   06:58   |   6 comments

TGen is breaking new ground in genomic research by using Dell's storage, cloud, and high-performance computing solutions.
On the Case
TGen IT: Where We Were

4|27|12   |   06:45   |   10 comments

The Translational Genomics Research Institute wanted to save lives, but its efforts were hobbled by immense computing challenges related to collecting, processing, sharing, and storing enormous amounts of data.
On the Case
1,200% Faster

4|18|12   |   02:27   |   12 comments

Through their partnership, Dell and TGen have increased the speed of TGen’s medical research by 1,200 percent.
On the Case
IT May Improve Children's Chances of Survival

4|17|12   |   02:12   |   8 comments

IT is helping medical researchers reach breakthroughs in a way and pace never seen before.
On the Case
Medical Advances in the Cloud

4|10|12   |   1:25   |   5 comments

TGen and Dell are pushing the boundaries of computing, and harnessing the power of the cloud to improve healthcare.
On the Case
TGen: Living the Mission

4|9|12   |   2:25   |   3 comments

TGen's CIO puts the organizational mission at the heart of everything the IT staff does.
On the Case
TGen Speeding Up Biomedical Research to Save More Lives

4|5|12   |   1:59   |   6 comments

The Translational Genomics Research Institute is revamping its computing to improve speed, storage, and collaboration – and, most importantly, to save lives.
On the Case
Computing Power Helping to Save Children's Lives

3|28|12   |   2:13   |   3 comments

The Translational Genomics Institute’s partnership with Dell is enabling them to treat kids with neuroblastoma more quickly and save more lives.
Tom Nolle
The Big Reason to Use Office

3|18|14   |   02:24   |   46 comments

Office and personal productivity tools come in a first-class and coach flavor set, but what makes the difference is primarily little things that most users won't encounter. What's the big issue in using something other than Office, and can you get around it?
E2 Editors
SPONSORED: Mobile Security — A Use Case

3|4|14   |   04:27   |   16 comments

New mobile security solutions can accommodate a wide array of needs, including those of a complex university environment.
Tom Nolle
Killing Net Neutrality Might Save You Money

1|16|14   |   2:13   |   16 comments

The DC Court of Appeals voided most of the Neutrality Order, and whatever it might mean for the Internet overall, it might mean better and cheaper Internet VPNs for businesses.
Tom Nolle
The Internet of Everythinguseful

1|10|14   |   2:18   |   19 comments

We really don't want an "Internet of Everything" but even building an Internet of Everythinguseful means setting some ground rules to insure there's value in the process and that costs and risks are minimized.
Tom Nolle
Maturing Google Chrome

12|30|13   |   2.18   |   25 comments

Google's Chrome OS has a lot of potential value and a lot of recent press, but it still needs something to make it more than a thin client. It needs cloud integration, it needs extended APIs via web services, and it needs to suck it up and support a hard drive.
Sara Peters
No More Cookie-Cutter IT

12|23|13   |   03.58   |   21 comments

Creating the right combination of technology, people, and processes for your IT organization is a lot like baking Christmas cookies.
Sara Peters
Smart Wigs Not a Smart Idea

12|5|13   |   3:01   |   46 comments

Sony is seeking a patent for wigs that contain computing devices.
Tom Nolle
Cloud in the Wild

12|4|13   |   02:23   |   15 comments

On a recent African trip I saw examples of the value of the cloud in developing nations, for educational and community development programs. We could build on this, but not only in developing economies, because these same programs are often under-supported even in first-world countries.
E2 Editors
SPONSORED: Is Malware Evading Your IPS?

11|18|13   |   03:16   |   4 comments

Intrusion prevention software is supposed to detect and block malware intrusions, but clever malware authors can evade your IPS in these five main ways.
Sara Peters
Where Have All the Mentors Gone?

9|27|13   |   3:15   |   38 comments

A good professional mentor can change your life for the better... but where do you find one?
Tom Nolle
SDN Wars & You Could Win

9|17|13   |   2:10   |   5 comments

VMware's debate with Cisco on SDN might finally create a fusion between an SDN view that's all about software and another that's all about network equipment. That would be good for every enterprise considering the cloud and SDN.
Ivan Schneider
The Future of the Smart Watch

9|12|13   |   3:19   |   39 comments

Wearing a bulky, oversized watch is good training for the next phase in wristwatches: the Internet-enabled, connected watch. Why the smartphone-tethered connected watch makes sense, plus Ivan demos an entirely new concept for the "smart watch."
Tom Nolle
Cutting Your Cloud Storage Costs

9|4|13   |   2:06   |   3 comments

Cloud storage costs are determined primarily by the rate at which files are changed and the possibility of concurrent access/update. If you can structure your storage use to optimize these factors you can cut costs, perhaps to zero.
Sara Peters
Do CIOs Need an IT Background?

8|29|13   |   2:11   |   23 comments

Most of the CIOs interviewed in the How to Become a CIO series did not start their careers as IT professionals. So is an IT background essential?
Ivan Schneider
The Internet Loves Birthdays

8|27|13   |   3:25   |   69 comments

The Internet has evolved into a machine for drumming up a chorus of "Happy Birthday" messages, from family, friends, friends of friends who you added on Facebook, random people that you circled on G+, and increasingly, automated bots. Enough already.