Knowing the limitations of big data

By Gaurav Gupta
Big data is feeding a revolution in how businesses make decisions and evaluate performance. Regular use of data and analytics to make decisions about bu...

Big data is feeding a revolution in how businesses make decisions and evaluate performance. Regular use of data and analytics to make decisions about business strategy and operations is now commonplace. Any operational metrics a business leader wants can be at their fingertips within seconds of it being requested. This can be a good thing – a great thing, even. 

Yet leaders need to be judicious in their use of data especially around privacy and sensitivity concerns. The well-known case of the department store Target using analysis of buyers’ shopping trends to market selectively to expectant mothers is a cautionary tale for leaders who want to rely on big data and analytics. While a great idea in theory, it backfired when a teenager who was hiding her pregnancy from her parents was called out by the coupons she received in the mail based on Target’s data analytics. 

See also:

A search for big data failures will reveal many such cases and each one of them stems from a reliance on data analytics without human intervention. The problem is not with big data and analytics but with how we choose to use them.

Another example is from the use of traffic light-based dashboards to review business performance. Hundreds of thousands of years of evolution have helped our brains develop a mechanism for spotting threats and problems. Our survival instincts cause us to want to eliminate threats. These mechanisms lead us to focus on the few ‘problem’ metrics that are invariably red, at the expense of potential upside opportunities represented by already ‘green’ metrics. 

Big data often leads managers to rely too much on the data and abdicate decision making. Using what the data tells you to inform a considered decision is of course commendable, but blindly following it without question or leaving room for experience and gut instincts can lead to poor decisions.  

In our effort to make what appear to be fact based and unbiased decisions we too often either ignore the uncertainty in our inputs or overstate the conclusions of our data analysis. Senior leaders need to set an example by ensuring that their teams can challenge the conclusions data lead towards, and talk openly about their own biases and opinions. Lots of research has shown the value of ‘gut feelings’ and the ability of the human brain to subconsciously process information and draw on experiences to arrive at intuitive decisions. The best decision making combines intuitive gut feeling with robust data analysis. 

The ever-increasing availability of data and the introduction of automation and artificial intelligence may seem to reduce the need for leadership and decision making, however the opposite is true. Big data and artificial intelligence allow us to ask ever more complex questions.  Making decisions that combine the answers they provide with intuition and experience will be the challenge for leaders. Data supports and amplifies the ability of people in a business to make well informed decisions and take action but should not be the sole driver for these decisions. 

The individuals, teams, and organisations that are most effectively able to harness the potential of big data will be those that are best able to integrate it into their existing decision-making methodology. 

Gaurav Gupta, Affiliate at Kotter


Featured Articles

How digital twins unlock enterprises’ sustainability efforts

With sustainability increasingly on corporate and government agendas, over half of enterprises believing digital twin technology is critical to ESG efforts

Avast: Cybercriminals use common apps to lure victims

Two out of three cyber threats now leverage social engineering, with attackers using common applications from Microsoft and Adobe to distribute malware

World Password Day: Study shows enthusiasm for passwordless

Over half of global respondents told a study that they are excited about passwordless authentication options like biometrics, passkeys, or security keys

SAP to accelerate AI innovation with IBM Watson

AI & Machine Learning

Half of organisations fell victim to ransomware attacks

Cloud & Cybersecurity

Nike and Cognizant expand their relationship into technology

Digital Transformation