IBM Release CodeFlare, a New Hybrid Cloud AI/ML Framework

CodeFlare’s goal is to simplify the integration and scaling of big data and AI workflows onto the hybrid cloud, IBM said

IBM Research has unveiled CodeFlare, a new framework for integrating and scaling big data and AI workflows in a hybrid cloud environment. The open-source framework aims to help developers cut back the time they spend creating pipelines to train and optimise machine learning models. 

CodeFlare is built on top of Ray, an emerging open-source distributed computing framework for machine learning applications. CodeFlare extends the capabilities of Ray by adding specific elements to make scaling workflows easier, according to IBM.

Researchers and developers have to train and optimise the model first to create a machine learning model today. CodeFlare simplifies this process using a Python-based interface for what’s called a pipeline—by making it simpler to integrate, parallelise and share data. The new framework aims to unify pipeline workflows across multiple platforms without requiring data scientists to learn a new workflow language.

 

A simpler way to integrate and scale full pipelines

 

CodeFlare pipelines run with ease on IBM’s new serverless platform IBM Cloud Code Engine, and Red Hat OpenShift, the company explained. It allows users to deploy it just about anywhere, extending the benefits of serverless to data scientists and AI researchers. It also makes it easier to integrate and bridge with other cloud-native ecosystems by providing adapters to event-triggers (such as the arrival of a new file), and load and partition data from a wide range of sources, such as cloud object storages, data lakes, and distributed filesystems.

CodeFlare "goes beyond isolated tasks to seamlessly integrate and scale end-to-end pipelines with a data-scientist-friendly interface--like Python--instead of using containers,'' said Priya Nagpurkar, director, hybrid cloud platform at IBM Research. "CodeFlare can provide a simpler way to integrate and scale full pipelines, while offering a unified runtime and programming interface."

The company has already seen CodeFlare in action and cutting time. For example, one user applied the framework to analyse and optimise approximately 100,000 pipelines for training machine learning models, CodeFlare cut the time it took to execute each pipeline from 4 hours to 15 minutes

Share

Featured Articles

Exec Q&A: Alex Cruz-Farmer, Cisco ThousandEyes

Alex Cruz-Farmer, Principal Product Manager at Cisco ThousandEyes, explains how their technology brings new levels of visibility to hidden DX issues.

Cloud & 5G - Day 2 highlights from the in-person stage

TECH LIVE LONDON returned to the Tobacco Dock last week. Stage host and Technology Magazine Editor in Chief, Alex Tuck, breaks down the presentations

Cloud & 5G - Day 1 highlights from the in-person stage

TECH LIVE LONDON returned to the Tobacco Dock last week. The stage host and Technology Magazine Editor in Chief, Alex Tuck, discusses the key themes

TECH LIVE LONDON: Day 2 highlights of the hybrid tech show

Digital Transformation

TECH LIVE LONDON: An overview of the hybrid technology show

Digital Transformation

TECH LIVE LONDON: Begins tomorrow at 10am!

Digital Transformation