CIOs and other IT leaders are increasingly being asked to work with colleagues across the organization to develop ways to mine structured and unstructured data in order to draw actionable business insights.
More data is available to enterprises than ever before, and analysis tools have never been cheaper. There's plenty of upside to undertaking these types of big-data projects, but there are also some big traps that can hurt your career and derail your funding for future projects.
Here are three procedural problems to avoid:
Putting the wrong people in charge of your big-data project
Big-data projects touch each part of your enterprise, requiring input, support, funding, and information from nearly every department. The resulting insights can ultimately benefit everyone in the company. Regardless of what it takes to get your numbers crunched and come up with clear answers, you need a strong leader for each big-data initiative. This leader might not be the person who automatically comes to mind.
Since the breadth and depth of data required is so large, big-data projects carry the potential for unprecedented scope creep. To avoid an endless string of "one more thing" bolt-on projects, your project leader needs to be firm, well respected, and operating with the very public support of executive management. Without these qualities, your project leader will flounder.
While the marketing department is probably your company's biggest consumer of data analysis, it's generally a mistake to put a pure marketer in charge of a big-data project. A CEB study quoted by the Harvard Business Review noted that many marketers are notoriously data averse. The handful of marketing executives who are data driven may veer too far in the opposite direction, over-relying on data over all other information sources.
You might be compelled to put a data scientist in charge. This is also ill advised, as these executives tend to lack the necessary political understanding to nurture relationships across the organization. They may also have difficulty handling the gut-level marketing requirements needed to produce useful end results.
Ideally, your big-data project leader should come from your company's program management department. This type of executive is the most likely to offer the necessary project management skills, far-reaching internal relationships, knowledge of the corporate culture, and an understanding of technology.
Letting your data analysis run amok
If analyzing big-data couldn't help your executives glean additional insights about customers and business, there wouldn't be much reason to do it. With that said, you can't go into a project without a clear focus, otherwise you'll be overwhelmed and distracted by too much information. Down the line, your analysis will pay off in unexpected ways, but that's a bonus. If you want your mining to generate a return, you have to start by looking to answer a specific question, or a very clearly defined set of questions.
Develop your focus and refine your queries first. Then stick to the plan. Are you trying to get a better handle on your true customer acquisition and costs? Are you looking at ways to reduce churn? Do you want to identify the prospects with the greatest likelihood of becoming customers? Focus on the primary questions first. After you've addressed these priority questions, you can process the exciting accidental discoveries that are sure to emerge during your analysis. Getting distracted chasing new tangents as they arise will only dilute your results and slow down your project.
Failing to anticipate dirty data
Your data is probably a lot dirtier than you think. Big-data implementations usually pull together multiple data sources that have never before been combined. In some cases, the data has never been analyzed, even on its own. De-duping your data, standardizing your formatting, and otherwise cleaning your data takes time. The smaller your test run, the faster you can work. After your data is clean, you can still expect to encounter procedural issues. Once again, working with small data sets allow you to tweak the system in a timely manner.
These are just some of the mistakes I've seen made by companies as they undertake big-data projects. I'd like to hear more about your experiences -- are they similar in nature? Are there other big-data pitfalls that come to mind? Tell us all about it in the comments section below.