Optimising Utilisation Forecasting with AI and Machine Learning
What IT team wouldn’t like to have a crystal ball that could predict the IT future, letting them fix application and infrastructure performance problems before they arise? Well, the current shortage of crystal balls makes the union of artificial intelligence (AI), machine learning (ML), and utilisation forecasting the next best thing for anticipating and avoiding issues that threaten the overall health and performance of all IT infrastructure components. The significance of AI has not been lost to organisations in the United Kingdom, with 43 per cent of them believing that AI will play a big role in their operations.
Utilisation forecasting is a technique that applies machine learning algorithms to produce daily usage forecasts for all utilisation volumes across CPUs, physical and virtual servers, disks, storage, bandwidth, and other network elements, enabling networking teams to manage resources proactively. This technique helps IT engineers and network admins prevent downtime caused by over-utilisation.
The AI/ML driving forecasting solution produces intelligent and reliable reports by taking advantage of the current availability of ample historic records and high-performance computing algorithms. Without AI/ML, utilisation forecasting relies on reactive monitoring. You set predefined thresholds for given metrics such as uptime, resource utilisation, network bandwidth, and hardware metrics like fan speed and device temperature. When a threshold is exceeded, an alert is issued. However, that reactive approach will not detect the anomalies that happen below that threshold and create other, indirect issues. Moreover, it will not tell you when you will need to upgrade your infrastructure based on current trends.
To forecast utilisation proactively, you need accurate algorithms that can analyze usage patterns and to detect anomalies—without false positives—in daily usage trends. That’s how you predict usage in the future. Let us take a look at a simple use case.
With proactive, AI/ML-driven utilisation forecasting, you can find a minor increase in your office bandwidth usage during the World Series, the FIFA World Cup, and other sporting events. That anomalous usage can be detected even if you have a huge amount of unused internet bandwidth. Similarly, proactive utilisation forecasting lets you know when to upgrade your infrastructure based on new recruitment and attrition rates.
A closer look at the predictive technologies reveals the fundamental difference between proactive and reactive forecasting. Without AI and ML, utilisation forecasting uses linear regression models to extrapolate and provide prediction based on existing data. This method involves no consideration of newly allocated memory or anomalies in utilisation patterns. Also, pattern recognition is a foreign concept. Although useful, linear regression models do not give IT admins complete visibility.
AI/ML-driven utilisation forecasting, on the other hand, uses the Seasonal and Trend decomposition using Loess (STL) method. STL lets you study the propagation and degradation of memory as well as analyze pattern matching whereby periodic changes in the metric configuration will be automatically adjusted. Bottom line, STL dramatically improves accuracy thanks to those dynamic, automated adjustments. And if any new memory is allocated, or if memory size is increased or decreased for the device, the prediction will change accordingly. This option was not possible with linear regression.
Beyond forecasting, ML can be used to improve anomaly detection. Here, adaptive thresholds for different metrics are established using ML and analysis of historic data will reveal any anomalies and trigger appropriate alerts. Other application and infrastructure monitoring functions will also be improved when enhanced with AI and ML technologies. Sometime in the not-too-distant future, AI/ML-driven forecasting and monitoring will rival the predictive powers of the fabled crystal ball.
by Rebecca D'Souza, Product Consultant, ManageEngine
AI Shows its Value; Governments Must Unleash its Potential
2020 has revealed just how far AI technology has come as it achieves fresh milestones in the fight against Covid-19. Google’s DeepMind helped predict the protein structure of the virus; AI-drive infectious disease tracker BlueDot spotted the novel coronavirus nine days before the World Health Organisation (WHO) first sounded the alarm. Just a decade ago, these feats were unfathomable.
Yet, we have only just scratched the surface of AI’s full potential. And it can’t be left to develop on its own. Governments must do more to put structures in place to advance the responsible growth of AI. They have a dual responsibility: fostering environments that enable innovation while ensuring the wider ethical and social implications are considered.
It is this balance that we are trying to achieve in the United Arab Emirates (UAE) to ensure government accelerates, rather than hinders, the development of AI. Just as every economy is transitioning at the moment, we see innovation as being vital to realising our vision for a post-oil economy. Our work in his space has highlighted three barriers in the government approach when it comes to realising AI’s potential.
First, addressing the issue of ignorance
While much time is dedicated to talking about the importance of AI, there simply isn’t enough understanding of where it’s useful and where it isn’t. There are a lot of challenges to rolling out AI technologies, both practically and ethically. However, those enacting the policies too often don’t fully understand the technology and its implications.
The Emirates is not exempt from this ignorance, but it is an issue we have been trying to address. Over the last few years, we have been running an AI diploma in partnership with Oxford University, teaching government officials the ethical implications of AI deployment. Our ambition is for every government ministry to have a diploma graduate, as it is essential to ensure policy decision-making is informed.
Second, moving away from the theoretical
While this grounding in the moral implications of AI is critical, it is important to go beyond the theoretical. It is vital that experimentation in AI is allowed to happen for its own sake and not let ethical problems stymie innovations that don’t yet exist. Indeed, many of these concerns – while well-founded – are born out in the practical deployment of these end-use cases and can’t be meaningfully discussed on paper.
If you take facial recognition as an example, looking at this issue in abstract quickly leads to discussions over privacy concerns with potential surveillance and intrusion by private companies or authorities’ regimes.
But what about the more specific issue of computer vision? Although part of the same field, the same moral quandaries do not arise, and the technology is already bearing fruit. In 2018, we developed an algorithmic solution that can be used in the detection and diagnosis of tuberculosis from chest X-rays. You can upload any image of a chest X-ray, and the system will identify if a person has the disease. Laws and regulations must be tailored to unique use-cases of AI, rather than lumping disparate fields together.
To create this culture that encourages experimentation, we launched the RegLab. It provides a safe and flexible legislation ecosystem to supports the utilisation of future technologies. This means we can actually see AI in practice before determining appropriate regulation, not the other way around. Regulation is vital to cap any unintended negative consequences of AI, but it should never be at the expense of innovation.
Finally, understanding the knock-on effects of AI
There needs to be a deeper, more nuanced understanding of AI’s wider impact. It is too easy to think the economic benefits and efficiency gains of AI must also come with negative social implications, particularly concern over job loss.
But with the right long-term government planning, it’s possible to have one without the other; to maximise the benefits and mitigate potential downsides. If people are appropriately trained in how to use or understand AI, the result is a future workforce capable of working alongside these technologies for the better – just as computers complement most people’s work today.
We’ve to start this training as soon as possible in the Emirates. Through our Ministry of Education, we have rolled out an education programme to start teaching children about AI as young as five years old. This includes coding skills and ethics, and we are carrying this right through to higher education with the Mohamed bin Zayed University of Artificial Intelligence set to welcome its first cohort in January. We hope to create future generations of talent that can work in harmony with AI for the betterment of society, not the detriment.
AI will inevitably become more pervasive in society, digitisation will continue in the wake of the pandemic, and in time we will see AI’s prominence grow. But governments have a responsibility to society to ensure that this growth is matched with the appropriate understanding of AI’s impacts. We must separate the hype from the practical solutions, and we must rigorously interrogate AI deployment and ensure that it used to enhance our existence. If governments can overcome these challenges and create the environments for AI to flourish, then we have a very exciting future ahead of us.