WeatherNext models are rigorously evaluated to demonstrate their skill and reliability compared to NWP methods. This evaluation primarily involves benchmarking against the European Centre for Medium-Range Weather Forecasts (ECMWF)'s leading operational systems, HRES (deterministic) and ENS (probabilistic). The models are extensively trained and assessed using ECMWF's ERA5 reanalysis data, which offers over 40 years of high-quality historical weather information, as well as ECMWF's Operational HRES -fc0 data from 2016.
Performance is measured using a suite of metrics, including Root Mean Squared Error (RMSE) for deterministic accuracy, Continuous Ranked Probability Score (CRPS) for probabilistic skill, and Anomaly Correlation Coefficient (ACC) for forecast verification. The Relative Economic Value (REV) is also used to quantify the practical benefit of probabilistic forecasts for extreme weather events. A key evaluation framework is WeatherBench 2, a standard benchmark developed by Google Research and ECMWF for assessing data-driven global weather models across various prediction ranges.
WeatherNext models provide more accurate, faster, and more reliable forecasts:
- WeatherNext 2 represents further advancement, demonstrating even greater skill than WeatherNext Gen across variables, forecast horizons and altitudes. The range of improvement tends to be around 10-20% better.
- WeatherNext 1: Gen, a probabilistic diffusion model, outperformed ECMWF's ENS, the leading operational ensemble forecast, over 96% of targets. The model excels in generating large ensembles to better characterize predictive uncertainty and provides superior performance in extreme event prediction, offering a lead time advantage of 6-12 hours for tropical cyclone tracking over ENS.
- WeatherNext 1: Graph, a deterministic model, outperformed ECMWF's HRES in 90% of tested cases and variables, showing particular skill in predicting tropical cyclones, atmospheric rivers, and extreme temperatures.
More details on the evaluation can be found in the publications. See the Research page.