Blending scientific expertise and technological prowess, the Met Office’s latest digital transformation is showing its peers how it’s done...
In the 60s, George Moore made a prediction that would set the pace of today's digital revolution. In a theory commonly dubbed as Moore’s Law, he forecast that the overall processing power of computers will double every two years or so. For the Met Office – the United Kingdom’s national weather service and climate agency – this prediction has proven to be a pertinent one. Processing vast amounts of climate and weather data from across the globe, Charles Ewen, CIO at the Met Office says that IT plays a critical role at the organisation, noting that his team aim to “provide world-class technology to support world-leading science”.
Without computers, weather forecasting as we know it would simply be impossible. Technology has played a pivotal role in understanding the weather for decades. In the 50s, the Met Office acquired its first electrical desk calculator and fast forward to today and the organisation have implemented the Cray XC40 supercomputer, or High-Performance Computer (HPC). At around 15 times the size of its predecessor, this mammoth machine is one of the most powerful supercomputers in the world, capable of completing 14,000trn calculations per second.
Ewen says that this supercomputer is instrumental in weather forecasting and climate prediction by making it possible to increase fidelity and resolution and provide more complete Earth System models. “The supercomputer contributes an awful lot because you can run models at finer resolutions with a lot more complexity,” he says. “They take into account more of the physics and chemistry that is involved and you can operate them over longer timescales and run simulations more frequently. We can also run the same model a number of times with some different assumptions and from that we can provide information about probabilities and likelihoods.”
Weather forecasting, Ewen explains, is an intricate balance of science and technology on a vast scale. At the Met Office, the organisation has been using innovations like data analytics for decades. “We’ve been working with data analytics at a large scale for many years although we typically don’t call these roles ‘data analysts’,” affirms Ewen. “We have chief meteorologists, for example, and their job is to look at what the computer simulations say and have some type of qualitative, contextual or impact assessment of what that data means. Our team have a unique blend of skills in data science, statistics, computer programming and more.”
The sheer size of the Met Office’s data pool cannot be overstated and as a result, the organisation shrewdly decided to shift to the cloud, partnering with technology giants such as Amazon Web Services, Google and Microsoft. “It’s still a journey,” reflects Ewen. “We certainly recognise that the move to the cloud gives us a combination of scale, elasticity, affordability and the capability that we need to support our world-class science teams.” Right now, the cloud does not fulfil the needs of its core simulations however as Ewen comments, “there is a lot of technology needed to convert those core simulations into the thousands of contextual forecasts and services that we produce every day”.
One of the most fundamental items in the Met Office’s toolkit is its simulations. Complex and state-of-the-art, this technology applies scientific knowledge to predict future atmospheric conditions across the globe using observations of the current weather captured from land, at sea, in the air, and from space. Ewen likens this to predicting a coin toss. Based on averages or statistics, you could contend that it’s a 50/50 chance that it will be either heads or tails, whereas the Met Office is trying accurately to simulate what side the coin will land on. “A statistical approach to weather forecasting doesn't give you the information you need because you want to know about anomalies, you want to know about extreme weather and you want to know about intensity. You don't want to know about averages. Essentially that is what the climate analysis tells you but a weather forecast is about the specifics of a given time and place.”
For many citizens, the weather forecast informs many day-to-day decisions like whether they should cut the grass tomorrow, what they should wear or how they should travel. It’s also vital for industries, allowing them to mitigate safety risks like preventing people from working outdoors in hazardous conditions. Emerging technologies like Artificial Intelligence (AI) offer the potential to take one step further. “I can be somewhat unpopular at work when I say that few people actually want a weather forecast,” Ewen says “more frequently, people or increasingly machines are trying to make a decision that is only partly about the weather. We are actively researching to understand how to better communicate the weather and future climate to these systems. Human beings are constrained in terms of how more statistics help better decision making, AI does not have that particular constraint,” he says.
Elsewhere, by applying Google Deep Mind, the office is hoping to research the realm of ‘nowcasting’ a form of weather prediction which uses radar returns and other observations to show how the weather may apply to more short-term localised regions. “If you watch this morning’s weather forecast and it says there’s likely to be heavy showers at 10 o’clock you might look out your window at that time to see no rain. It might be a very near miss spatially; it might rain in 20 minutes but not exactly at 10 o’clock. Nowcasting, as opposed to forecasting, applies statistics to see how the weather may evolve in the short term.”
Whilst technology plays a key role in the Met Office’s digital transformation roadmap, Ewen asserts that it’s the people that are at the crux of its success. As such, a key part of the organisation’s strategy revolves around people. “Within our strategy we have a plan to attract retain and develop our people and so it’s clearly very important. We talk about mastery, autonomy and purpose. Mastery refers to developing the unique blend of skills that we need so we invest strongly in training our people,” he says. Indeed, Met Office has developed a variety of new apprenticeships and works to broaden its relationship with academic settings. “We also try to create an exciting environment that people want to work in,” he adds.
The Met Office has worked hard to forge a work culture which strives to do more. The biggest barrier to continuous improvement? Sparking a cultural shift whereby people recognise the need to change. “That can be tricky when you’re a world leading organisation at the top of its game,” admits Ewen. “However, because we’re an organisation full of bright and able people we can see where some of the future challenges and opportunities lie. Because of that, it's important to start to change ideally ahead of when you really need to, which is what we've been trying to do.”
One example of this lies at the Met Offices fingertips – its supercomputer. As Moore’s Law dictates, computer processing power will only continue to expand over the coming years and the Met Office needs to be ready to tap into the potential of future supercomputers. “Because the of scale our operations are so large, it can be difficult to predict what the technology environment will look like in the future. Take supercomputers, for example, we’re always looking five to ten years in the future. It’s important to realise what you need to change now, to be lucid and clear about what the target destination looks like.”
By all accounts, cutting-edge science and technology are the bread and butter of the weather forecasting. Whether it's going to be blustery or raining, the blend of the two means that we can predict the weather more accurately than ever. “Every ten years, the weather forecast has improved with the same accuracy being available a day earlier,” concludes Ewen. “In other words, a four-day weather forecast is as accurate as a three-day forecast was ten years ago. That shows the rate of progress and technology undoubtedly plays an important part of that.”