With so many big-data projects now underway across the enterprise, the next big challenge facing IT organizations is how to effectively manage that amount of data.
Big-data governance has suddenly become one of the hottest topics in IT as organizations turn their attention once again to the fundamentals of data management; except this time, it’s at a scale no one previously thought possible, or, for that matter, advisable.
Establishing big-data governance policies is critically important because the amount of data involved not only threatens to overwhelm IT organizations, but how that data gets used will actually determine the success of such projects. Specifically, organizations need to understand what type of data is valuable enough to keep as opposed to data that is expendable.
Unfortunately, a recent survey of 3,500 IT leaders conducted by TEKsystems, a provider of IT services, found that 66 percent of IT leaders and 53 percent of IT professionals said their data is stored in disparate systems and that they need new platforms to accommodate these increased data management needs.
In addition, 81 percent of IT leaders said their organization lacks the skills on staff to plan, build, and run big-data initiatives.
That’s problematic because a separate survey of 1,000 Fortune 1,000 business and IT leaders conducted by NewVantage Partners found that half the executives surveyed said they would be spending more than $10 million on a big-data project by 2016.
"Companies are appointing chief data officers to drive innovation," Randy Bean, founder and managing partner for NewVantage Partners, told Enterprise Efficiency. "That means there’s also multiple tiers of governance with a lot of active IT participation."
Bean added that while certain types of big-data projects involving -- for example -- compliance, are defensive in nature, the majority of the investments being made in big-data these days are of an offensive nature. Knowing the difference between those classes of projects requires a lot of context.
For example, speaking at recent IBM Software event in New York, Michele Goetz, a senior analyst for Forrester Research, said rather than simply storing data for data’s sake simply because it’s now more affordable to do so using platforms such as Hadoop, organizations need to think through what data ultimately needs to be saved.
“All data is not created equal,” Goetz said. “Context is critical; governance needs to be a lot more about how you actually manage the business than the data.”
Speaking at the same event, Heather Wilson, chief data officer for AIG, added the real issue goes beyond simple governance.
“The business has to have confidence in the data,” said Wilson. “That requires discipline.”
“There’s definitely a governance gap when it comes to big-data,” Henry Olson, director of product management for Embarcadero Technologies, a provider of big data tools, told us. "We think organizations will need to leverage metadata to close the gap."
Whatever the approach, IT organizations need to prepare to manage not only vast amounts of data at levels of scale previously thought unimaginable, but also an array of new big-data types that are increasingly finding their way into the enterprise.