Why the way businesses deal with data is changing

A solid data strategy is crucial to the digital transformation of businesses, however many businesses fail to implement effective data projects. Research from Gartner suggests that 85% of data science projects fail to go into production or generate any value. Furthermore, IDG research suggests that only one in three data science projects succeed, even with advancements in the application of AI. The ability to leverage data will be the key differentiator that will enable businesses to thrive, but how can businesses improve the success rate of data science projects?

With new job roles, technologies and organisational cultures changing the way that businesses deal with data, business leaders can extract more value from the data that is available to them. There is a critical need for business leaders to train internal talent in basic data skills to become data experts that can transform their business’s ability to derive insights from data. This will also prevent the need to rely on an over-stretched central data team that outputs business insights into the form of reports and visualisations.

Bridging the gap between IT and the data consumers

Many businesses are bringing in data experts to leverage data initiatives due to their ability to interpret high volumes of data and turn it into actionable insights. However, relying solely on data experts can be a crucial error for businesses, due to the obstacles facing teams that cannot be dealt with alone. This can lead to a breakdown between artificial divisions of data experts and business users, with data experts becoming more business-minded and business users learning to ‘self-serve’ with data. One aspect of this is the rise of roles such as ‘analytics engineer’, which help to bridge the gap between IT and data consumers within an organisation. Analytics engineers collaborate with the team to analyse the data, to ensure that the business can use the high-quality insights generated from their work. Together with wider teams, these engineers help to set up and activate a truly modern data stack. 

Training employees to become data citizens

Rather than relying solely on hiring qualified data experts, business leaders should aim to train their existing workers with data skills: this can help to keep costs and overheads down. Data literacy courses are already becoming common in many companies, and large organisations such as Bloomberg and Adobe are going further, with in-house digital academies dedicated to training workers in how to use data.

Training analysts to use low-code or no-code tools for data management costs far less than hiring a data scientist. By removing bottlenecks in daily data operations, teams that need analytic dashboards to make decisions for campaigns don’t have to wait and can focus more on revenue-generating activities. 

 Training existing employees is particularly powerful because they combine newly acquired data skills with their existing domain expertise to extract maximum value from the data. These ‘data citizens’ will be able to extract value from data without waiting for a separate team of data experts or scientists to do it for them. 

Unlocking the value of data through technology

Democratising access to data within your organisation and unlocking the business value of data requires the right technological tools. Data management is an important tool to ensure data is delivered to the right team within your business, in a condition where it can be used, without the bottlenecks and delays that can come from relying on a central data team. 

Data management deployments automate procedures into one framework, making it simpler for business users to extract value. Along with tools such as data quality management, data validation ensures that data meets the standards required by business users.

Perhaps even more important is Reverse ETL, which turns the normal job of data warehouses on their head to direct a stream of valuable data directly to the teams which need it most. Reverse ETL reverses the traditional process by which data is loaded into a data warehouse, by first extracting it from a data warehouse and then loading it into your operational systems.

In Reverse ETL, the data is loaded from the data warehouse and then fed directly into business software such as ERP (Enterprise Resource Planning) or CRM (Customer Relationship Management). Sales or marketing teams have data delivered directly into the applications they use in their daily work, meaning there’s less training required to understand it. 

For example, this can be used to deliver personalised offers based on purchase history or more precisely targeted marketing campaigns. It’s key to breaking down the barriers between data and the data consumers within a company, and removing a burden from overworked specialist data teams.

Enabling data management, through data mesh 

Along with these technological changes and job role evolutions around data, there is also a new organisational approach to how data works within companies; a data mesh. In short, data mesh offers a decentralised and ‘self-serve’ approach to delivering data throughout an organisation. Rather than relying on a centralised data team – where the warehouse is controlled by hyper-specialised experts – data is organised via shared protocols, in order to serve the business users who need it most.

The significance of this is that it helps empower teams to access the correct data they need, right when they require it, via the distribution of data ownership across the organisation. Whilst the concept of data mesh isn’t necessarily new, the key to operationalising this approach effectively is the introduction of a platform or universal interoperability layer that facilitates the connection of domains and associated data assets within it. Companies can then use a platform that will help them to connect all the dots and manage the entire operation, in order to fully operationalise the approach.

Being aware of the business value of data is no longer enough. Companies need to start adopting a “data as a product” approach as data mesh will be the core to enabling the application of the product life cycle to data deliverables. By applying product thinking to datasets, a data mesh approach will ensure that the discoverability, security and explorability of datasets are retained. Teams are then better prepared to swiftly derive the most important insights from their data.

The growth of data citizens 

It is important to consider that central data teams can become a bottleneck if analysts and engineers across the business can’t access the data they need, when they need it. Therefore, getting the right access for the right people is essential for businesses to make timely decisions. Data management teams should work in collaboration with other teams to provide users with the skills and tools they need to self-serve and have the ability to access data within the systems and processes that they’re already using. This will ensure that businesses can evolve to have data citizens throughout the company, without the need for hyper-specialisation. Through having an internal team of data experts in place across a majority of functions, teams can act on the data in real-time and make key decisions that ensure they will not be left behind.