Business Intelligence practitioners should steal some lessons from the marketing department in their quest to bring data and analytics to the masses. Instead of obsessing with improving existing technology, BI promoters should focus on what is needed at the Top of the Funnel and find new ways to make those users productive.
It is widely accepted that BI systems are under-used. The penetration of Business Intelligence into the enterprise, depending on how you count it, is about 30 percent, meaning less than a third the people in a business actually use data and analytics to do their jobs. Analysts who make such estimates often point to deficiencies in user experience and high complexity as the main culprits.
BI is Stuck at the Bottom of the Funnel
BI systems in use now mostly represent what I call Bottom of the Funnel technology. The funnel is a concept used by marketing to explain the journey of a user in their understanding of a technology and also to organize the different ways of attracting them.
Top of the Funnel users are those just looking around. Marketers attract them with content of broad interest to get as many people engaged as possible. Then through a series of “nurturing touches”, that is offers to interact with content, people express interest. In response, marketers send different types of content that provides more detailed information. If they interact with that content it is assumed they are getting serious about making a decision and want more information. When people exhibit buying signals, such as using an ROI calculator or taking some detailed product tour, then they get sent the most detailed content and are passed to the sales staff.
The reason that most BI represents Bottom of the Funnel technology is that it is created for a specific context. A lot is known about the end-user and what they want, just like at the bottom of a marketing funnel. Dashboards to support a specific role in a business process, advanced analytical environments, complicated multi-sheet spreadsheets, and thick reports all fall into the bottom of the funnel category. They are super helpful for a small segment of the population.
What is Top of the Funnel BI?
We can easily create a working definition of Top of the Funnel BI by building on this way of thinking. Top of the Funnel BI would allow people to interact with the system and express their interests, and then get answers.
The perfect Top of the Funnel, aka as TOFU, BI would be like HAL in 2001: A Space Odyssey. You could just ask questions and get answers. We are a long way from that sort of experience, but some huge progress has been made in the past few years. At the most ornate level is IBM’s Watson, of course, but we must be able to do more with less to really solve the TOFU problem.
But the popular presentation of HAL and Watson misses something important that must be included in TOFU BI, the idea of supporting an exploration that helps you decide want you want to know. Donald Farmer, VP of Product Management of QlikView, pointed out to me for a natural language system to work optimally, it must incorporate some form of lexical entrainment to help disambiguate language (“Here is the dashboard for net income, click gross income to get a dashboard for that quantity”) and also suggest better questions (“Do you want to compare that to the same period last year?”). In other words, TOFU BI will have to support an exploratory conversation to reveal what Quentin Clark, Corporate Vice President, Data Platform Group, Microsoft, calls, the “BI intent”, that is the answer that a person is after.
My view is that the success of QlikView and Tableau is partially due to the fact that these systems take a big step toward TOFU BI by supporting both exploration and refinement of questions. Each instance of a QlikView or Tableau powered dashboard can answer a large amount of questions. Here’s a simplified way think of this:
- Reports may answer 5 or 10 questions about a topic.
- A single spreadsheet based dashboard may answer 10 or 20.
- A configurable dashboard or a multi-spreadsheet system may answer as many as 50 questions.
- One well-designed environment in QlikView or Tableau may answer as many as 100 questions.
QlikView and Tableau answer more questions than the systems more toward the BOFU because they can incorporate more data into an environment that can be explored. Tableau does a great job of presenting queries in a visually compelling fashion that can be explored in a super intuitive way. QlikView provides the ability to show the relationships between many data sets and to express incredibly complex queries and results in a simple and intuitive way that includes visualizations. QlikView’s associative model also shows you what data did not match your query, which it turns out is an unexpectedly powerful feature.
QlikView and Tableau dashboards incorporate more data because both of them have advanced ways of modeling and connecting large amounts of information. QlikView’s associative data model and Tableau’s VizQL both make it easier for the dashboard designer to bring together large amounts of information. It is important to understand that the increased amount of information translates into the power to answer more questions and make sense of more data. Even though the semantics of a QlikView or Tableau dashboards are generally inside the mind of the user and not explicitly supported, these dashboards succeed because they have the power to enable end-users to find more semantic relationships, in other words to make sense of the data.
In an indirect way, I assert that the success of both of these companies is that they accelerate users ability to create new semantic models of data. For TOFU BI to really take off, the systems directly use semantic models.
Natural Language is Required for Making TOFU BI a Success/>/>
We are now facing a world in which the amount of data available is dramatically exploding. The most complete TOFU BI will not require a designer to collect a set of information and then craft it to meet the needs of a specific context, which is how almost all BOFU BI systems are built.
For TOFU BI to work at a minimal level, the following requirements must be satisfied:
- The end-user must express their desire using natural language and the system must then create from that language a model of the question being asked,
- That model of the question must be connected to data that can answer that question.
- The system should automatically create a display of the answer.
- The system should make suggestions to the user for how the answer can be extended and refined.
As I pointed out in “How Semantics Can Make Data Analysis Work Like Google Search”, DataRPM has many elements of this in place. Microsoft PowerBI for Office 365 is also exploring what can be done with queries expressed in natural language. Such systems are in their early days, but the best ones already work well enough so it is clear they will get better as the ability to automatically create semantic models of data and connect them to both natural language and to visualizations improves.
The promise of systems that meet the TOFU BI requirements set forth above is that the dashboards created will be able to answer hundreds or even thousands of questions. It is likely that TOFI BI will still be targeted toward a broad segment of users who are interested in asking questions about a defined domain of data. But unlike BOFU BI, the data and the visualizations will come to the user instead of having to be explicitly designed. Of course, at first the dashboard created by TOFU BI systems will be simpler than those created for BOFU BI. But it is likely that at the TOFU level, the questions will be simpler as well.
- Even though technologies like QlikView and Tableau and others have increased the potential for self-service BI, people don’t want to spend their time creating BI systems, they want to do their jobs.
- The amount of data that is relevant to almost any job is growing too fast for anyone to keep track of.
TOFU BI will succeed because it will automate what is now being done manually for BOFU BI. To return to the marketing analogy, what TOFU BI does is make each natural language asked the equivalent of a nurturing touch. The system then responds with data, dashboards, and suggestions for refinements and then uses the interaction with the users to create a better answer and to provide more information in a useful form. As the TOFU BI system is used, the user proceeds toward a more complete answer.
The key questions are: How quickly will the vendors pursuing this vision be able to get it right? How much of this vision has to work to provide significant value? For companies who would get a big boost from expanding use of data an analytics, it is time to start the experiments.
Follow Dan Woods on Twitter:
Dan Woods is CTO and editor of CITO Research, a publication that seeks to advance the craft of technology leadership. For more stories like this one visit www.CITOResearch.com. Dan has performed research for QlikView and Tableau.