Why are Big Data, Predictive Analytics and Data Visualization Projects So Difficult?

Everywhere you look in the press people are talking about Big Data, AI, Machine Learning and the potential they have to transform your business. These articles talk about a tool or company that makes the process sound so easy. It is not unlike opening a fashion magazine where everyone is skinny, wrinkle-free and beautifully dressed. However, the deeper you dig, the more you see that not everyone is that perfect and that Big Data projects do not have the success and adoption rates that many companies had envisioned.

Why are these projects so difficult?

Reason 1: Your expectations are screwed up (So are senior management’s)

In the press you are bombarded by messages from companies selling tools that help with a particular aspect of Big Data, Predictive Analytics and Data Visualization. According to the advertising messages these tools will solve all your woes and are easier to use than ever. To extend the fashion metaphor…”Lose 15 Lbs. in a week” “Get a Six-Pack in 15 minutes a day”. These companies are advertising. They are going to stress how they simplify aspects of the project and do their best not to focus on other areas that may be uglier/dirtier in fear that they may discourage your investment.

Set realistic expectations: Managing and mining large disparate data sets frequently requires a team of people with functional excellence across multiple disciplines. It requires, a methodical, efficient, repeatable process. It requires some of the tools we mentioned above. And it requires a boatload of change management within your organization to action the insights. That is not, and will never be, “easy”. Don’t go into your project assuming a tool alone will solve your problems. Realistic expectations are a great place to start.

Reason #2: Getting your hands on the data is not a walk in the park:

The Big Data projects that we work on typically involve bringing together data from a company’s internal IT team (ERP/CRM/Etc.), advertising and digital agencies, social media sites, 3rd party data syndicators, government sources, etc. Before you can analyze or visualize this data, you need to get your hands on it. That means identifying the data owners, obtaining permissions, auditing the content/formats/systems and getting it all in one secure location. For smaller projects, this may be less overhead, but when your project requires feeds from many sources, within and outside your organization, setting aside time to build an inventory of the data will help you plan for and resource the effort.

Reason 3: The value and effort behind harmonizing the data is misunderstood:

Harmonizing Data

Let’s say you have access to all of the data. You have even brought the data together in one location within your data center or on Amazon’s (AWS) or Microsoft’s (Azure) cloud. Can we start building the models and visualizing the data? No. The census data is at the DMA/quarter level, the advertising data is at the region/week level, the digital data is national and has not been mapped to specific brands and the social stats are real-time. By far the least understood element of these projects is the level of effort required to harmonize the data. What does harmonize mean? It means transforming the data to align the dimensions (products, geographies, time, etc.) so that all of the different sources of information can be compared and analyzed together. Think about it as ensuring that the “who”, “what”, “where” and “when” behind the data are all linked and aligned.

Reason 4: You need to source and then automate (hide) the analytics:

So you brought together the data in one place, figured out how to harmonize the data and gave your insights team access to the resulting analytic data set on your Data Lake. They then built a really cool Bayesian Model using R or Python and generated awesome insights. You want this model to be leveraged to deliver insights to your end users (Brand Managers, Field Sales Reps, etc.) on a continuous basis. Do you set them all up in R (or SAS or SPSS) and teach them how to run the models? Clearly that is not the right direction. So how do you build a process and tool set that takes the models outputs and delivers those insights without requiring end-users to pay licenses and understand the analytics and tools used to generate those models? You need to figure out a way to componentize the great work from your analytic team. Put it in a module that can be called from your reporting and visualization tools without requiring deep analytic knowledge or expensive licensing expansion.

Reason 5: You need to streamline delivery to each group of end users (and they all want something different):

Have you ever read articles by Stephen Few? A great source of information on how to visualize data to facilitate understanding. But the challenge is bigger than just how to layout the data. What information do you share and what is not needed? How do you display it? How do you give users access to it? Is it via PowerPoint in their email? Is it via a dashboard via a website? Is it through Tableau or PowerBI on the user’s desktops/tablets/phones? The funny thing is that there is no one “right” answer. Your content, layout and delivery strategies need to be tailored to each audience. Do you have the right platforms to support multiple content, layout and delivery vehicle combinations?

Reason 6: You need to have your end users change the way they think about the business and utilize data

Let’s face it…change management is tough. In fact, it may be the hardest part of the entire process. You could have successfully addressed all of the issues above and you still have one big hurdle. Getting your end users to change the way they do their jobs. Want to maximize adoption? Think about a project roadmap where you reduce the amount of change required by your end users in the early phases of the project. Make their current lives easier, get them excited about the new solutions and then introduce new capabilities and expanded usage scenarios as their comfort grows.

Conclusion:

Self-service visualization. Self-service analytics. These terms are thrown around a lot in today’s publications. To be clear, the arsenal of tools we have today is more powerful, and easier to use than ever. However, that does not mean you can license a tool for $20 a month and solve all of your issues. Thinking about some of the challenges that we have outlined above, may help you proactively address these hurdles before they negatively impact your project.

We hope this helps!

---

WholeStory helps Sales and Marketing teams leverage data and analytics to make better decisions. We deliver practical solutions that harmonize data, execute advanced analytics, and maintain performance dashboards and reporting systems. For more information, please reach out to us @ (203) 800-4151 or email: info@wholestory.io