MIT and Snowflake Spotlight Data Engineering’s AI Impact

In a world accelerating into an AI future, data engineers are fuelling the race.

New research from the MIT Technology Review, conducted in partnership with Snowflake, reveals that 72% of technology leaders believe data engineers to be integral to the overall success of their business. 

In organisations with annual revenues exceeding US$10bn, this figure rises to 86%.

“Senior leadership is quickly realising that if they’re not using data in their decision-making processes, they risk lagging behind their competitors,” says Chris Child, Vice President of Product Data Engineering at Snowflake.

“This has really brought data engineers to the attention of senior business leaders, with data engineers increasingly serving as essential partners for shaping strategy and driving business outcomes.”

Data engineering expectations rise, but so does the workload

As AI integrates deeper into organisations, traditional data engineers are expected to increase their efficiency, says Chris. 

“For one thing, the amount of data that AI models require is growing exponentially, as is the number of AI projects.

“But it’s more than volume," he says. “Data engineers are managing more complexity, such as unstructured data and real-time pipelines. 

“They’re also managing the ever-increasing expectations of business stakeholders.”

Research shows that data engineers now spend 37% of their time working on AI projects, a jump from 19% two years ago. 

This is expected to reach 61% by 2027, making data engineers a cornerstone of AI transformation within organisations. 

According to the research, 77% respondents said that there is a significant increase in workload among data engineers, who now play a key role as AI enablers in their companies.

Dave Masino, Senior Director of Data and Intelligence at technology consulting company Slalom, echoes this point. 

“AI has increased the amount of work data engineers are doing, but you inherently want your team to be busy,” he says.

“I see the advantage we’re getting from AI-enabled acceleration eclipsing the amount of additional workload.”

Agentic AI changes the data engineering game

One-fifth of the organisations that participated in the research are already working with agentic AI models. 

More than half, 54%, say they will follow along the agentic AI route within the next 12 months.

For data engineers, agentic AI offers respite from tedious tasks while improving their productivity and output.

Chris explains: “We’ll start to see more agentic data engineering where having [AI] agents do a larger chunk of their operational work allows data engineers and teams to think about the bigger picture.

“They’ll ask, ‘What are our overarching goals? What budget do I give to which of these agents to process data? How do we think about our overall data estate rather than just individual pipelines?'

“Then you’ll start to see a larger shift in the role of the data engineers.”

This change would mean that data engineers could evolve within organisations to take on more strategic roles.  

AI-powered data engineering

In the past two years, 74% of respondents reported improved data engineering team productivity with the use of AI, in terms of the quantity of work delivered.

Quality has also significantly improved, with 77% citing a notable boost in the team's output. 

Dave says: “With advancements in Gen AI over the past two to three years and, most importantly, its integration into software development tooling, data engineers now have a very powerful accelerator at their disposal.”

AI powered tools now automate and assist data engineers with data cleansing, integration, pipeline monitoring, metadata management, workflow orchestration, feature engineering and other critical tasks. 

The research shows that today's data engineers must not only master AI, but also develop business acumen, communication and presentation skills to deliver real organisational value.