How to Build a Quality Assurance Framework for AI and ML

AI regularly proves its value, so it’s natural to want to extend its scope and role. Consider self-driving cars. Today, driver-assist features work well across entire fleets of automobiles, inspiring confidence in their future. The aim is to make real-life interactions with pedestrians, motorists and cyclists significantly safer with intelligent, judicious and unflinching decision-making processes. 

When advanced AI processes a vehicle’s sensor data, it improves the vehicle’s automatic decision-making. However, recent studies have shown that without the right framework, the level of safety might be unreliable. If the AI’s quality or underlying systems are flawed, it could make decisions that lead to catastrophe and endanger lives, such as issuing a “turn right” or “go straight” instruction instead of “turn left.” 

Creating a framework to solve these challenges sounds straightforward. It’s not. One issue impacting the quality of AI models is the high variability of skills and knowledge among AI developers, coupled with the lack of a standardised approach to AI. 

The confluence of these issues results in sub-par AI models that could deliver questionable outcomes and degrade AI’s trustworthiness, reliability, and dependability. As a result, the USA and EU are introducing bills that would make organizations liable for lapses in AI. Consequently, the move toward a standard framework and quality assurance for AI and ML models is paramount.

Six Essential Principles for a Quality Assurance Framework

An assessment framework can evaluate AI systems on quality and ethical parameters and certify a “Quality Assured AI model.” A quality assurance framework should include six quality zones: 

• Functional Suitability denotes the completeness, correctness and accuracy of the AI model
• Efficiency will entail the resource requirements and execution simplicity of the AI model 
• Portability is the ease AI models switch between algorithms
• Maintainability measures the ease of switching signals in the model and testing for the appropriate ones 
• Security for AI models checks for data poisoning and leakage in the ML data pipelines
• Usability pinpoints the transparency and ability to explain the AI model 

A quality assessment framework designed for AI models should measure, test and certify models to be of standard quality based on each quality zone. AI processes are an amalgamation of several lifecycle stages, are interdependent and serve significantly different purposes. At each lifecycle stage, quality assessments should test the interdependent processes. 

Breaking Down the Quality Assurance Best Practice Framework

The best practice framework should facilitate the development of quality AI models and identify and use relevant KPIs. In each stage, the activity revolves around measuring the quality of a lifecycle stage, comparing it with pre-set benchmarks, and designing a framework to ensure adherence to quality and ethical standards. 

Specification Collection

Start by understanding business pain areas, identifying Critical-To-Quality (CTQ) parameters, and converting the problems to optimizing the relevant operational process. 

Data Acquisition and Pre-Processing

The data science team identifies and collates data sources while mending data quality issues. Depending on the use case, the CTQs, and identified operational parameters in the previous stage, database experts devise processes to collate datasets from multiple sources and offer augmentation tactics with AI and domain experts.

Exploratory Data Analysis

An AI team needs to conduct a deep scan of the data to identify relationships between data points and resolve issues of outliers and missing information. Properties of the data and multi-variable relationship mapping can be excavated by skilful use of statistics and employing a business-centric explanation of data for better understanding. 

Feature Engineering

Feature engineering is churning out data signals from datasets that pinpoint hidden data patterns. A deep dataset scan is essential to find features that implicate patterns. The framework should allow data scientists to build, experiment and store relevant generic, reusable and scalable feature-creation recipes. 

Model Development and Tuning

The AI team will produce powerful ML routines to equate any event to the occurrence of data signals. Moreover, the team will probe the compliance of developed ML models and select champion models from the list of candidates. Standard statistical modelling performance parameters will verify model-building efficacy. 

Model Deployment and Monitoring

AI developers and engineers will inspect deployed assets and identify slipping performance that requires re-training. Testing deployed assets using statistical KPIs will identify data drift, population stability and deviation from normalcy. The framework will use MLOps to provide a generalised approach for ML pipeline creation and includes constant pipeline monitoring to identify data discrepancies and model poisoning. 

The Way Forward for AI Quality

The lack of a comprehensive quality assurance framework can lead to rogue AI/ML models that can hamper enterprise processes and continue to erode confidence in AI models. AI solutions need a standardized framework that measures and imparts quality and ethics in the models. For enterprises in the EU, the consequences of the new AI act begin in 2023, so the time for a quality assurance framework is now.