OceanBench: The Sea Surface Height Edition

J. Emmanuel Johnson, Quentin Febvre, Anastasiia Gorbunova, Sammy Metref, Maxime Ballarotta, Julien Le Sommer, Ronan FabletNeurIPS 2023

The ocean is the Earth's heat sink and a primary driver of our climate, yet we only observe a tiny fraction of it at any given time. Satellite sensors provide a "bird's-eye view" of sea surface height (SSH), but due to orbital paths and sensor limitations, these observations are incredibly sparse—covering less than 5% of the globe daily. Filling these gaps is a classic "inverse problem" ripe for machine learning (ML), but the barrier to entry for AI researchers remains notoriously high.

In a new paper presented at NeurIPS 2023, researchers introduce OceanBench, a framework designed to turn messy satellite data into ML-ready benchmarks.

Executive Summary

OceanBench is an open-source framework that provides standardized datasets and "plug-and-play" pipelines for sea surface height (SSH) interpolation. By embedding domain-specific geoprocessing into a modern MLOps workflow, it allows ML researchers to tackle complex oceanographic problems without needing a deep background in geoscience.

The Challenge of Ocean Monitoring

Monitoring the ocean is a data-starved endeavor. While satellites measure SSH, the resulting data is "holey." Imagine trying to reconstruct a high-definition movie when you only have 5% of the pixels, scattered irregularly across the screen.

Traditionally, geoscientists have relied on "Optimal Interpolation" (kriging) or complex physical models to fill these gaps. While ML has shown massive potential here, the pre-processing phase is a nightmare for outsiders. Researchers must navigate coordinate transformations, irregular sampling, and noise levels that vary by sensor. Furthermore, standard ML metrics like Mean Squared Error (MSE) are often insufficient for oceanography; a model might achieve a low MSE but fail entirely to capture the actual physics of ocean eddies (the "weather" of the sea).

Methods: The OceanBench Framework

OceanBench is not just a static dataset; it is a comprehensive co-design framework. It connects domain-specific data preparation with modern deep learning ecosystems.

OceanBench Framework Architecture

The architecture consists of three main pillars:

  1. OceanBench-Data-Registry: Utilizes Data Version Control (DVC) to manage and share large, ML-ready datasets efficiently.
  2. Ocean-Tools: A suite of geoprocessing functions (built on xarray) that handles domain-heavy lifting, such as regridding, filtering, and coordinate shifts.
  3. xrpatcher: A bridging tool that translates geographical data structures into arbitrary "patches" suitable for training in deep learning frameworks like PyTorch, TensorFlow, or JAX.

The framework leverages Hydra for configuration management, allowing researchers to easily swap out variables, regions, or model architectures while maintaining a strictly reproducible pipeline.

Datasets and Challenges

The authors focus their initial release on the Gulf Stream region (a 1000km x 1000km area) and provide four distinct challenge tracks:

  • OSSE NADIR: Synthetic data simulating traditional "along-track" satellite observations.
  • OSSE SWOT: Simulates the new, high-resolution "wide-swath" altimetry from the SWOT satellite mission.
  • OSSE SST: A multi-modal challenge that combines SSH with Sea Surface Temperature (SST) data.
  • OSE NADIR: Real-world satellite observations, providing a "true" test of how well models generalize from simulation to reality.

The synthetic datasets (OSSE) are derived from the NEMO ocean model, which provides a reliable "ground truth" to measure model performance against.

Key Insights

  • The "Physics-Aware" Metric: The paper emphasizes that researchers shouldn't rely solely on standard error scores. OceanBench incorporates Spectral Analytics (Isotropic Power Spectrum) to determine the smallest physical scales a model can actually "resolve." If a model smooths out all the small eddies, it might have a great RMSE but remain scientifically useless.
  • Multi-Modal Synergy: A major takeaway is the utility of Sea Surface Temperature. SST data is often available at a much higher resolution than SSH. OceanBench makes it easy to use SST as a "guide" to help models infer where SSH structures should be.
  • Standardization as an Accelerator: By providing a front-end for data selection and a back-end for physical evaluation, the authors argue the community can move from "toy" problems to real-world impact much faster.

Key Results

The paper provides benchmarks for several model classes, including coordinate-based models (like MIOST) and end-to-end variational schemes (like 4DVarNet).

  • 4DVarNet emerged as a top performer, achieving a normalized RMSE of 0.95 and resolving spatial scales down to 117km (a significant improvement over the 175km resolution of traditional methods).
  • The results demonstrate a clear hierarchy: models that incorporate physical "priors" or multi-modal data (like SST) significantly outperform "blind" interpolation methods.

Limitations and Future Work

While OceanBench represents a major step forward, it currently focuses on a single region (the Gulf Stream) and a single variable (SSH). The authors acknowledge that the "generalization gap"—where a model trained on simulations fails on real-world data—remains a persistent challenge.

Future iterations of OceanBench aim to expand to global scales and include other critical variables like ocean currents, salinity, and carbon uptake. The ultimate goal is to pave the way for a "foundation model" approach for the world's oceans.

Resources: * Framework: github.com/jejjohnson/oceanbench * Data Registry: github.com/quentinf00/oceanbench-data-registry

Summary updated

BibTeX
@inproceedings{johnson2023oceanbench,
  author = {J. Emmanuel Johnson and Quentin Febvre and Anastasiia Gorbunova and Sammy Metref and Maxime Ballarotta and Julien Le Sommer and Ronan Fablet},
  title = {{OceanBench: The Sea Surface Height Edition}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
}
Figure