Red Hat Steps Into AI Optimisation with Neural Magic Deal

The challenge of efficiently deploying AI systems has emerged as a critical issue for enterprises. As companies rush to implement large language models, they face mounting infrastructure costs and technical complexities in running these systems at scale. The computing power required to operate AI models has grown exponentially, with recent models demanding hundreds of times more resources than their predecessors of just two years ago.

Against this backdrop, Red Hat, the enterprise Linux software provider owned by IBM, has moved to strengthen its position in AI infrastructure. The company has announced its agreement to acquire Neural Magic, a Massachusetts Institute of Technology spinout that develops software to improve the performance of generative AI (Gen AI) systems.

The deal aims to address a specific challenge in AI deployment: the process of inference, where AI models generate responses to queries. Neural Magic, founded in 2018, has developed software that helps large language models run more efficiently across different types of computer hardware, potentially reducing the need for specialised and expensive AI accelerators.

Red Hat, which provides enterprise Linux software and cloud computing platforms to more than 90% of Fortune 500 companies, sees Neural Magic's technology as central to its strategy of helping organisations run AI workloads in different computing environments, from corporate data centres to cloud platforms.

Integration with open-source project

A core element of the acquisition centres on Neural Magic’s involvement with vLLM, an open-source project developed at the University of California Berkeley. The project provides software for serving AI models across different hardware platforms, including processors from AMD, Intel and Nvidia, as well as custom chips from Amazon Web Services and Google.

“AI workloads need to run wherever customer data lives across the hybrid cloud,” says Matt Hicks, President and CEO of Red Hat. “This makes flexible, standardised and open platforms and tools a necessity, as they enable organisations to select the environments, resources and architectures that best align with their unique operational and data needs.”

The technology will complement Red Hat’s existing AI portfolio, which includes Red Hat Enterprise Linux AI, a platform for running IBM’s open-source Granite language models, and Red Hat OpenShift AI, which provides tools for developing AI systems using the Kubernetes container orchestration software.

Hardware flexibility and cost efficiency

The acquisition addresses growing demand for software that can run AI models across different computing environments. Neural Magic’s technology helps companies optimise their AI systems for both cloud computing facilities and on-premise data centres, potentially reducing the need for specialised and expensive AI hardware.

The company’s software focuses on inference – the process where AI models generate responses to queries. This stage of AI deployment presents particular challenges for organisations, as it requires balancing performance with cost considerations across different types of computing hardware.

Brian Stevens, Chief Executive Officer of Neural Magic, says: “We've assembled some of the industry's top talent in AI performance engineering with a singular mission of building open, cross-platform, ultra-efficient LLM serving capabilities. Joining Red Hat is not only a cultural match, but will benefit companies large and small in their AI transformation journeys.”

Industry implications

The deal marks a significant move in the enterprise software sector’s response to growing AI adoption. Red Hat’s acquisition provides a path for organisations to deploy AI systems without being locked into specific hardware platforms or cloud providers.

The companies have not disclosed the terms of the acquisition, which requires regulatory approval. The transaction comes as technology providers race to provide infrastructure for running AI models, which require substantial computing resources.

For IBM, which has made AI central to its corporate strategy, the acquisition strengthens its ability to serve clients across different computing environments. “As our clients look to scale AI across their hybrid environments – virtualised, cloud-native LLMs built on open foundations will become the industry standard," says Dario Gil, IBM Senior Vice President and Director of Research. “Red Hat’s leadership in open source combined with the choice of efficient, open source models like IBM Granite and Neural Magic’s offerings for scaling AI across platforms empower businesses with the control and flexibility that they need to deploy AI across the enterprise.”

Make sure you check out the latest edition of Technology Magazine and also sign up to our global conference series - Tech & AI LIVE 2024

Technology Magazine is a BizClik brand