IBM Release CodeFlare, a New Hybrid Cloud AI/ML Framework

CodeFlare’s goal is to simplify the integration and scaling of big data and AI workflows onto the hybrid cloud, IBM said

IBM Research has unveiled CodeFlare, a new framework for integrating and scaling big data and AI workflows in a hybrid cloud environment. The open-source framework aims to help developers cut back the time they spend creating pipelines to train and optimise machine learning models. 

CodeFlare is built on top of Ray, an emerging open-source distributed computing framework for machine learning applications. CodeFlare extends the capabilities of Ray by adding specific elements to make scaling workflows easier, according to IBM.

Researchers and developers have to train and optimise the model first to create a machine learning model today. CodeFlare simplifies this process using a Python-based interface for what’s called a pipeline—by making it simpler to integrate, parallelise and share data. The new framework aims to unify pipeline workflows across multiple platforms without requiring data scientists to learn a new workflow language.

 

A simpler way to integrate and scale full pipelines

 

CodeFlare pipelines run with ease on IBM’s new serverless platform IBM Cloud Code Engine, and Red Hat OpenShift, the company explained. It allows users to deploy it just about anywhere, extending the benefits of serverless to data scientists and AI researchers. It also makes it easier to integrate and bridge with other cloud-native ecosystems by providing adapters to event-triggers (such as the arrival of a new file), and load and partition data from a wide range of sources, such as cloud object storages, data lakes, and distributed filesystems.

CodeFlare "goes beyond isolated tasks to seamlessly integrate and scale end-to-end pipelines with a data-scientist-friendly interface--like Python--instead of using containers,'' said Priya Nagpurkar, director, hybrid cloud platform at IBM Research. "CodeFlare can provide a simpler way to integrate and scale full pipelines, while offering a unified runtime and programming interface."

The company has already seen CodeFlare in action and cutting time. For example, one user applied the framework to analyse and optimise approximately 100,000 pipelines for training machine learning models, CodeFlare cut the time it took to execute each pipeline from 4 hours to 15 minutes

Share

Featured Articles

NTT Brings AI and Data Innovation to the Indianapolis 500

NTT is harnessing cutting-edge technologies like AI and data analytics to revolutionise how the Indianapolis 500 race is competed and experienced by fans

Salesforce & IBM Partnership to Drive AI, Data Deployment

IBM and Salesforce's expansion of their partnership shows how watsonx’s is making inroads in enterprises across sectors

FC Barcelona & Fortinet: Cybersecurity Takes Centre Stage

Fortinet is deploying its Security Fabric platform as part of a partnership with FC Barcelona, aimed at providing cutting-edge cybersecurity

Google Cloud Generative AI Ops Drives Enterprise AI Adoption

AI & Machine Learning

How Publicis Sapient Helps Your Digital Transformation

Digital Transformation

How Google Cloud is Providing a Launchpad for Women in Tech

Cloud & Cybersecurity