DataOps: Bridging the divide
Technology leaders can’t go ten minutes without being bombarded with the latest propaganda pitching a solution to their digital transformation challenges. While the marketing is tired, the problem is not. Leaders everywhere are struggling to innovate at high velocity. It’s no longer the big that are eating the small, but the fast that are eating the slow.
The digital economy has been defined by the proliferation of computing through mobile, web, and IoT, all enabled by tech such as DevOps and cloud. But if the past is about speed, the future is about data. With the rise of connected consumer technology and advances in machine learning and artificial intelligence, data, and the insights we can derive from it, are the foundation of tomorrow’s economy.
Despite the obvious value, companies struggle to realise the opportunity lying dormant in their data. When developers, testers and data scientists have access to the data they need, innovation happens faster. But data is more complex, more unwieldy, and harder to secure. Legacy tools, organisations, and processes can’t resolve this tension - we need a new approach.
DataOps is an emerging cultural movement that aligns people, process, and technology to support high velocity collaboration and innovation through data. By focusing on bringing people together to secure, manage, and deliver data, DataOps practices can enable data to flow wherever it’s needed by the business.
But what actually is DataOps?
Like DevOps, DataOps is broad in its approach across people and technology. DataOps is about aligning tools, processes, and culture to facilitate secure access of data, regardless of how data may be used across existing teams or organisations. Its goal is to improve outcomes by facilitating access to data and collaboration between the teams that use data.
DataOps brings together everyone that is involved in the creation, management, and use of data. This includes those that operate data infrastructure such as IT, DBA, and InfoSec teams. But it also includes those that use data to drive new projects and innovation, such as developers, testers, analytics, and data scientists.
If these teams exist today, why do we need a new approach to bring them together?
The rising demand for data creates natural tension with the forces trying to manage cost, complexity, and security. This tension creates a divide between the teams and tools trying to provide people access to the data they need. Higher quality data drives better outcomes, and this divide can force compromises that put the business at risk. For example, if a developer needs access to the latest production data to test a new change, their demand creates friction on the systems and processes required to fulfil that request. If that request takes days or weeks, the team is forced to decide between moving slower, or risking poor application quality due to inadequate data.
This problem is only getting harder. The rise of artificial intelligence and machine learning increases demand for large, representative datasets. Cloud and DevOps brings elastic application environments to the masses but struggles to fill them with data, and the proliferation of voice, image, and video data require new tools and techniques.
DataOps is as much about transformation of people as it is about transformation of technology. To be successful, companies need to tear down the walls that have been created over time and instead take a data-centric view of their enterprise, one that brings together everyone that manages, or needs access to, data.
Start by mapping out the data you have (or want to have), and where it’s needed. Think through not just application development, but analytics and data science, archive and compliance, and cloud migration. From those data needs, articulate the policies that need to be in place to ensure that only the right people get access to the right data in the right environment. For example, what data is subject to HIPAA or GDPR? What data can the public cloud support versus a private data centre?
By bringing everyone together into the conversation and focusing on the business policies and needs of data users, companies can map out how data should flow across their organisations and infrastructure. This will expose constraints and points of friction that can be mitigated or worked around through new technology or processes. DataOps solutions should see to put self-service access to data in the hands of users, while empowering IT teams with confidence that data is flowing according to the policies they’ve established.
A happy ending with DataOps
Global revenues from artificial intelligence (AI) for enterprise applications is projected to grow from £1.14 billion in 2018 to £22.3 billion in 2025, with 49% of companies in 2017 using some form of marketing automation. However, your AI is only as good as the data you feed it, and much of the most importance data remains trapped behind traditional enterprise systems and processes.
With DataOps, operations can focus on establishing and securing all the data sources needed by the project teams, without being bombarded with a plethora of requests for access from all over their organisations. Meanwhile, developers are equipped with fresh and secure production data which can be easily refreshed, reset and shared. The result is faster projects, of higher quality, at a lower cost.
It’s time to stop thinking about digital transformation and start embracing data transformation through DataOps.
Eric Schrock, Chief Technology Officer at Delphix
SAS: Improving the British Army’s decision making with data
SAS’ long-standing relationship with the British Army is built on mutual respect and grounded by a reciprocal understanding of each others’ capabilities, strengths, and weaknesses. Roderick Crawford, VP and Country GM for SAS UKI, states that the company’s thorough grasp of the defence sector makes it an ideal partner for the Army as it undergoes its own digital transformation.
“Major General Jon Cole told us that he wanted to enable better, faster decision-making in order to improve operational efficiency,” he explains. Therefore, SAS’ task was to help the British Army realise the “significant potential” of data through the use of artificial intelligence (AI) to automate tasks and conduct complex analysis.
In 2020, the Army invested in the SAS ‘Viya platform’ as an overture to embarking on its new digital roadmap. The goal was to deliver a new way of working that enabled agility, flexibility, faster deployment, and reduced risk and cost: “SAS put a commercial framework in place to free the Army of limits in terms of their access to our tech capabilities.”
Doing so was important not just in terms of facilitating faster innovation but also, in Crawford’s words, to “connect the unconnected.” This means structuring data in a simultaneously secure and accessible manner for all skill levels, from analysts to data engineers and military commanders. The result is that analytics and decision-making that drives innovation and increases collaboration.
Crawford also highlights the importance of the SAS platform’s open nature, “General Cole was very clear that the Army wanted a way to work with other data and analytics tools such as Python. We allow them to do that, but with improved governance and faster delivery capabilities.”
SAS realises that collaboration is at the heart of a strong partnership and has been closely developing a long-term roadmap with the Army. “Although we're separate organisations, we come together to work effectively as one,” says Crawford. “Companies usually find it very easy to partner with SAS because we're a very open, honest, and people-based business by nature.”
With digital technology itself changing with great regularity, it’s safe to imagine that SAS’ own relationship with the Army will become even closer and more diverse. As SAS assists it in enhancing its operational readiness and providing its commanders with a secure view of key data points, Crawford is certain that the company will have a continually valuable role to play.
“As warfare moves into what we might call ‘the grey-zone’, the need to understand, decide, and act on complex information streams and diverse sources has never been more important. AI, computer vision and natural language processing are technologies that we hope to exploit over the next three to five years in conjunction with the Army.”
Fundamentally, data analytics is a tool for gaining valuable insights and expediting the delivery of outcomes. The goal of the two parties’ partnership, concludes Crawford, will be to reach the point where both access to data and decision-making can be performed qualitatively and in real-time.
“SAS is absolutely delighted to have this relationship with the British Army, and across the MOD. It’s a great privilege to be part of the armed forces covenant.”