This page explains how to access and use WeatherNext forecasts stored in Google Cloud Storage (GCS). Storing data on GCS is ideal for users who need direct, bulk access to the forecast files for use in custom workflows, cloud computing environments, or applications that can read the Zarr format directly.
GCS is particularly well-suited for:
- High-performance computing: Directly access data from cloud-based virtual machines or distributed computing frameworks like Dask.
- Maximum flexibility: Use any programming language or library that can interact with GCS and the Zarr data format.
- Bulk data access: Efficiently work with large portions of the dataset without needing an intermediary query service.
A quick introduction to Zarr
The WeatherNext forecasts on GCS are stored in the Zarr format. If you're new to Zarr, here's what you need to know:
- N-dimensional Arrays: Zarr is designed to store N-dimensional arrays, like the multi-variable, gridded data found in weather forecasts (e.g., latitude, longitude, time, variable).
- Chunked Storage: The data is broken into smaller "chunks." This is the key to its efficiency. When you access a slice of the data (like a small geographic area or a single point in time), you only need to download the relevant chunks, not the entire massive dataset.
- Parallel I/O: The chunked format allows for many chunks to be read or written in parallel, which is excellent for performance in cloud environments.
- Community Standard: Zarr is a widely used, open standard in the scientific community, with strong support in Python via libraries like Xarray.
Available datasets and how to access
The forecast data is available in the following GCS buckets:
- WeatherNext 2.0:
gs://weathernext/weathernext_2_0_0/zarr - WeatherNext Gen:
gs://weathernext/126478713_1_0/zarr - WeatherNext Graph:
gs://weathernext/59572747_4_0/zarr
To access the datasets, you will need to fill out the WeatherNext Data Request form.
Starter guide and sample code
For an in-depth tutorial and sample code, check out the WeatherNext 2 Starter Guide - Zarr on Google Cloud Storage notebook.
Things to be aware of
Terms of use
- Historical Data (forecasts more than 48 hours old): Licensed under the Creative Commons Attribution International License, Version 4.0 (CC BY 4.0).
- Real-Time Data (forecasts generated within past 48 hours): Made available under the GDM Real-Time Weather Forecasting Experimental Data Terms of Use.
Citations
If you use this data in your research, be sure to cite the appropriate papers and datasets. You can find the citation information on the dataset pages.