Why automation will be vital for next generation video games

Automation is having an increasingly transformative effect on our lives, from robotic process automation in the workplace to AI assistants in our phones to automated driving features in our cars. It’s no surprise, then, that automation technologies are set to also leave their mark on the next generation of video game consoles.

With the two big beasts Sony and Microsoft scheduled to release the latest versions of their PlayStation and Xbox video game consoles by the end of the year, both are looking to use automation to add value to their respective offerings. That’s partly enabled by the increased power the consoles now have.

In just one example, Sony has filed patents for dynamic sound creation, using machine learning to combine different musical components into music that fits the tone of the game at any particular moment. Microsoft, meanwhile, is including machine learning algorithms to generate high dynamic range (HDR) content from SDR content.

Beyond the consoles, automation is increasingly being brought in to augment and assist in the development of games themselves.

One such example of this is the latest entry in Microsoft’s long-running Flight Simulator series, the first entry of which debuted in 1982. Comparing that first edition to the latest, the leap in fidelity achieved in the intervening 40-year period is obvious. Gone are the blocky lines and solid colours, replaced by a virtual representation of reality perfectly capable of fooling people into believing it is real (save for the occasional glitch).

Achieving that level of detail across the entire globe, as the latest edition does, should be well beyond the efforts of any game studio - no matter how large - were it not for the deployment of AI to fill in the gaps. Asobo Studios, the game’s developers, have duly partnered with a number of technology providers within and without publisher Microsoft’s umbrella.

Austrian tech company Blackshark.ai’s contribution came in the form of using AI to extract infrastructure information from satellite imagery, in order to construct a digital twin of the planet’s buildings. Digital twin technology is most-often deployed in an industrial or construction setting, rather than the traditionally more bespoke practice of game development, transferring physical environments into more digestible digital versions.

Digital twins

• The digital twin market will be worth $16bn by 2023
• Digital twins are most popular in “asset-heavy” sectors such as the oil and gas, aerospace and automotive industries
• Outside of industry, digital twins are increasingly popular for retail, healthcare and smart city purposes

Source: Deloitte

We’ve previously spoken to PGS Software, a provider of digital twin software, who explained to us what the technology involves. “A digital twin is just a visual replica of a physical object or process or product, and it's usually updated in real time in order to match reality,” said Maciej Mazur, Chief Data Scientist. “This is, in my opinion, the best way to connect the physical world with the digital realm. Thanks to the fact that you can have digital twins at different levels, from a single sensor to a machine, you can make the whole production line one big digital twin composed of the smaller levels. There’s a lot of freedom to emulate and use machine learning algorithms to improve production.”

In a video, the company described the challenge of extending the photogrammetric coverage of the world’s buildings, vegetation and infrastructure from 0.2% to 100%. “Blackshark.ai has implemented a unique solution that uses the Microsoft Azure Cloud and artificial intelligence to gain insights about our planet. Based on Bing Maps imagery data, a deep learning neural network segments and classifies buildings, vegetation and roads globally. A patented approach is used to reconstruct detected building attributes in highly detailed 3D.” 

Stefan Habenschuss. Head of the Machine Learning Group, added: “Over the course of the last three years we've reconstructed approximately 1.5 billion buildings and detected over 30 million square kilometers of vegetation. This just proves the scalability and performance of the technology we used. Artificial intelligence is filling in the gaps from limited input data, achieving stunning results that indeed make the player feel like they're flying over the real world. While building footprints, height type and rooftops are being segmented by a neural network with high accuracy, additional attributes such as facade features are added automatically based on geographical and contextual data.”

While the gaming industry expected to be worth $300bn by 2025, the largest tech companies are increasingly investigating the space, using their technical clout to introduce new solutions such as this form of automation. Microsoft has previously suggested that their greatest rivals in the gaming space are no longer Sony and Nintendo, but other big tech firms with the capacity to match their endeavours.

What's the video game market worth?

• Video game market worth $18.4bn in the US in 2017
• Consumer spending on gaming in the US reached $43.4bn in 2018

Source: Statista

It makes sense, then, that Microsoft is also bringing its cloud expertise to bear on the game. As well as using AI, Microsoft Flight Simulator relies on Microsoft's Azure cloud platform, into which algorithms and data were fed to create a 2.5-petabyte model including 2 trillion trees, 1.5 billion buildings and 117 million lakes. With no chance of such a vast volume of data being stored locally, the game instead relies on streaming it in to players on the fly. Competitors such as Google have also levied their cloud computing power, albeit in Google’s case it was for the streaming of games entirely via its Stadia platform.

As the ambition of those in the video game industry continually extends, from console manufacturers to video game developers, emerging technologies are filling in the gaps. From cloud systems processing volumes of data infeasible in a home setting, to AI generating ever more impressive facsimiles of reality, the future is automated.