Western Digital Explains AI Workload & the AI Data Cycle

As AI continues to evolve, the interplay between data generation and storage has become increasingly crucial.

AI technologies permeate various industry sectors and they are not only consuming vast amounts of existing data but also generating new data at an unprecedented rate.

This continuous loop of data consumption and creation is reshaping the requirements for data storage infrastructure, presenting both challenges and opportunities for organisations across the globe.

The concept of the AI Data Cycle has emerged as a framework for understanding the complex relationship between AI and data storage.

This cycle encompasses six distinct stages, each with its own unique storage requirements. As AI continues to advance, storage solutions must evolve to meet the demands of these various stages, balancing performance, capacity, and cost-effectiveness.

To find out more, we asked Peter Hayles, Product Marketing Manager for HDDs at Western Digital, about the implications of AI on data storage and the intricacies of the AI Data Cycle.

The AI data cycle: explained

Peter begins by emphasising the transformative nature of generative AI and its impact on data storage.

"There is no doubt that generative AI is the next transformational technology," he states.

"As AI technologies become embedded across virtually every industry sector, inspiring a world of new applications, storage is becoming an increasingly important and dynamic component of the AI technology stack."

He goes on to explain the continuous loop of AI data, where increased data generation fuels expanded data storage, which in turn fuels further data generation.

This cyclical process underscores the critical role that storage plays in the AI ecosystem.

The stages of the AI data cycle

Peter outlines the stages of the AI Data Cycle, each with its own specific storage requirements:

Data collection and storage
In this initial stage, raw data is collected and securely stored from various sources.
Peter notes: "The quality and diversity of collected data is key, setting the foundation for everything that follows."
He recommends capacity enterprise hard disk drives (eHDDs) for this stage, citing their ability to provide "lowest cost bulk data storage, delivering highest capacity per drive and lowest cost per bit."

Data processing and preparation
The second stage involves processing, cleaning, and transforming data for input to model training.
Peter explains that data centre owners are implementing upgraded storage infrastructure to support this stage, including "fast data lakes to support preparation and ingestion."

He adds that "all-flash storage systems incorporating high-capacity enterprise solid state drives (eSSDs) are deployed to augment existing HDD based repositories, or within new all-flash storage tiers."

Model training
During the third stage, AI models are trained iteratively on high-performance supercomputers.
Peter emphasises the importance of efficiency in this stage, stating: "Efficiency relies heavily on maximising GPU utilisation.
“As a result, very high-bandwidth flash storage near the training server is important, with high-performance (PCIe® Gen. 5) and low-latency compute optimised eSSDs designed to meet these stringent requirements."

The evolution of AI infrastructure

As AI continues to transform, so too must the infrastructure supporting it.

Peter highlights the need for organisations to maintain current systems alongside new AI compute, driving further storage needs.

"AI models will be integrated into existing internet and client applications, enhancing them without replacing current systems," he explains.

"This means maintaining current systems alongside new AI compute, driving further storage needs."

This integration will require upgrades to existing storage systems, including additional data centre eHDD and eSSD capacity, as well as larger and higher performance client SSDs (cSSDs) for PCs and laptops, and higher capacity embedded flash devices for mobile phones, IoT systems, and automotive applications.

The final stages: deployment and content creation

Peter describes the fifth stage as "where the magic happens in real-time," involving the deployment of trained models into production environments.

He notes that organisations might deploy "high-capacity eSSDs for streaming context or model data to inference servers, and, depending on scale or response time targets, high-performance compute eSSDs for caching."

The final stage of the AI Data Cycle involves the creation of new content.

Peter explains: "The insights produced by the AI models often generate new data, stored because it proves valuable or engaging.

While this stage closes the loop, it also feeds back into the data cycle, driving continuous improvement and innovation by increasing the value of data for training or analysis by future models."

Looking to the future

As we look to the future of AI and data storage, Peter anticipates continued growth and innovation.

"The continuous loop of data generation and consumption is accelerating the need for high-capacity and performance-driven scalable storage technologies for managing large AI data sets and re-factoring complex data efficiently, driving further innovation," he states.

Peter concludes by predicting that: "storage component providers [will] increasingly tailor products to the needs of each stage in the cycle," highlighting the ongoing importance of efficient data storage in the AI ecosystem.

As AI continues to reshape industries and drive innovation, understanding and optimising the AI Data Cycle will be crucial for organisations looking to harness the full potential of this transformative technology.

The evolution of data storage solutions will play a pivotal role in supporting the next generation of AI applications and services.

******

Make sure you check out the latest edition of Technology Magazine and also sign up to our global conference series - Tech & AI LIVE 2024

******

Technology Magazine is a BizClik brand