Knowing the limitations of big data

Big data is feeding a revolution in how businesses make decisions and evaluate performance. Regular use of data and analytics to make decisions about business strategy and operations is now commonplace. Any operational metrics a business leader wants can be at their fingertips within seconds of it being requested. This can be a good thing – a great thing, even. 

Yet leaders need to be judicious in their use of data especially around privacy and sensitivity concerns. The well-known case of the department store Target using analysis of buyers’ shopping trends to market selectively to expectant mothers is a cautionary tale for leaders who want to rely on big data and analytics. While a great idea in theory, it backfired when a teenager who was hiding her pregnancy from her parents was called out by the coupons she received in the mail based on Target’s data analytics. 

See also:

• Cloud optimisation: Overcoming the complexities of cloud-based solutions

• Vantage Data Centers raises $1.1bn to fund two US construction projects

• Encryption: The good and the bad

A search for big data failures will reveal many such cases and each one of them stems from a reliance on data analytics without human intervention. The problem is not with big data and analytics but with how we choose to use them.

Another example is from the use of traffic light-based dashboards to review business performance. Hundreds of thousands of years of evolution have helped our brains develop a mechanism for spotting threats and problems. Our survival instincts cause us to want to eliminate threats. These mechanisms lead us to focus on the few ‘problem’ metrics that are invariably red, at the expense of potential upside opportunities represented by already ‘green’ metrics. 

Big data often leads managers to rely too much on the data and abdicate decision making. Using what the data tells you to inform a considered decision is of course commendable, but blindly following it without question or leaving room for experience and gut instincts can lead to poor decisions.  

In our effort to make what appear to be fact based and unbiased decisions we too often either ignore the uncertainty in our inputs or overstate the conclusions of our data analysis. Senior leaders need to set an example by ensuring that their teams can challenge the conclusions data lead towards, and talk openly about their own biases and opinions. Lots of research has shown the value of ‘gut feelings’ and the ability of the human brain to subconsciously process information and draw on experiences to arrive at intuitive decisions. The best decision making combines intuitive gut feeling with robust data analysis. 

The ever-increasing availability of data and the introduction of automation and artificial intelligence may seem to reduce the need for leadership and decision making, however the opposite is true. Big data and artificial intelligence allow us to ask ever more complex questions.  Making decisions that combine the answers they provide with intuition and experience will be the challenge for leaders. Data supports and amplifies the ability of people in a business to make well informed decisions and take action but should not be the sole driver for these decisions. 

The individuals, teams, and organisations that are most effectively able to harness the potential of big data will be those that are best able to integrate it into their existing decision-making methodology. 

Gaurav Gupta, Affiliate at Kotter