AutoML for Climate Change: A Call to Action

Renbo Tu, Nicholas Roberts, Vishak Prasad, Sibasis Nayak, Paarth Jain, Frederic Sala, Ganesh Ramakrishnan, Ameet Talwalkar, Willie Neiswanger, Colin WhitearXiv 2022 · DOI · OpenReview

TL;DR

This paper investigates whether off-the-shelf Automated Machine Learning (AutoML) tools can automatically discover high-performing models for critical climate change AI (CCAI) applications, specifically climate modeling, wind power forecasting, and catalyst discovery. The authors found that current AutoML libraries, largely designed for computer vision and NLP, do not significantly outperform human-designed models in these domains, highlighting a need for specialized AutoML search spaces.

What Problem Does This Paper Address?

Climate change is a pressing global issue, and Artificial Intelligence (AI) is increasingly being leveraged to address it through Climate Change AI (CCAI). CCAI tackles complex problems often involving physics-constrained machine learning or heterogeneous spatiotemporal data. The promise of Automated Machine Learning (AutoML) is to automate the process of finding optimal model architectures and hyperparameters, which could be particularly beneficial in these specialized CCAI domains where human expertise might be scarce or the problem space is vast. However, it's unclear if existing, general-purpose AutoML tools, which are heavily influenced by computer vision (CV) and natural language processing (NLP) paradigms, can effectively handle the unique challenges of CCAI. This paper aims to bridge this gap by evaluating the efficacy of current AutoML techniques on key CCAI tasks and identifying areas for improvement.

Methods

The authors benchmarked popular AutoML libraries, specifically Optuna for hyperparameter optimization (HPO) and SMAC3 for neural architecture search (NAS). They applied these tools to three distinct CCAI tasks:

  • Climate Modeling: Using the ClimART dataset, they performed HPO on a CNN baseline and NAS across various architectures (MLP, CNN, GNN, GCN, L-GCN).
  • Wind Power Forecasting: For the SDWPF dataset, they conducted HPO on BERT-based and GRU+LGBoost models, and NAS on the BERT-based model.
  • Catalyst Discovery: They used the OC20 dataset and applied HPO to a Graphormer model.

The core methodology involved comparing the performance of models found by AutoML with existing, human-designed baseline models on each task. The researchers explicitly aimed to test AutoML "out-of-the-box" performance, meaning they did not spend time designing custom search spaces tailored to the specific CCAI tasks.

Datasets

The paper utilizes three established datasets for its experiments:

  • ClimART: A dataset for simulating atmospheric radiative transfer, providing global snapshots of atmospheric data from 1979 to 2014. It was used in the NeurIPS Datasets and Benchmarks Track 2021.
  • SDWPF (Spatial Dynamic Wind Power Forecasting): A large dataset featuring data from 134 wind turbines over 12 months, sampled every 10 minutes. This dataset was recently featured in a KDD Cup 2022 competition.
  • OC20 (Open Catalyst 2020): A dataset for catalyst discovery, focusing on predicting the energy required for chemical reactions. The paper used a down-sampled version of the OC20 IS2RE task.

Key Insights and Ideas

  • AutoML's Current Limitations in CCAI: The most significant insight is that readily available AutoML tools, as they stand, do not consistently outperform human-designed models in these high-impact CCAI applications. This suggests that the current AutoML landscape is not yet optimized for the unique challenges of climate science.
  • The "CV/NLP Bias" in AutoML: A key weakness identified is that most AutoML techniques and their associated search spaces are heavily tailored towards computer vision and NLP tasks. This lack of specialization hinders their effectiveness in domains like spatiotemporal forecasting or physics-constrained modeling.
  • Need for Domain-Specific Search Spaces: The paper strongly advocates for the development of AutoML search spaces that are specifically designed for CCAI challenges. For instance, search spaces that can effectively interpolate between different types of neural network architectures (MLPs, CNNs, GNNs, GCNs) or incorporate spatiotemporal data structures are crucial.
  • Physics Constraints as a Future Frontier: Many CCAI problems inherently involve physics laws. The paper highlights that incorporating these physics constraints into AutoML search spaces and loss functions is a promising avenue for future research, potentially leading to more accurate and physically consistent models.

Key Results

  • No Significant Improvement: Across the three CCAI tasks (climate modeling, wind power forecasting, and catalyst discovery), out-of-the-box AutoML techniques generally failed to achieve performance significantly better than human-designed models.
  • ClimART Exception (with caveats): For the ClimART dataset, AutoML showed some improvement. However, the authors note that they were unable to reproduce the original reported performance of the human-designed CNN model, and the AutoML performance was compared against their own (worse) evaluation of the default model.
  • Time Investment: The AutoML experiments required substantial computational resources, running for 12-50 hours on each task.

Limitations and Future Work

Limitations: * The experiments were not exhaustive; only one trial was run per experiment due to computational constraints. * The AutoML methods were used "out-of-the-box" without custom-designed search spaces tailored to the CCAI tasks. * The hyperparameter ranges were based on default values and popularity, not an exhaustive exploration.

Future Work: * Designing Domain-Specific Search Spaces: This is identified as the most critical area. Developing search spaces for spatiotemporal forecasting and for incorporating physics constraints is paramount. * Exploring Other AutoML Sub-areas: Beyond HPO and NAS, future work could investigate AutoML for data augmentation, data preprocessing, and continuous model monitoring. * Broader CCAI Application Testing: The authors suggest testing AutoML on a wider range of CCAI applications, such as model predictive control for buildings and optimal power flow. * Responsible AI Development: The paper emphasizes the need for responsible and climate-conscious research in AutoML for CCAI, considering the environmental and financial costs of training large models.

Summary updated

Abstract

The challenge that climate change poses to humanity has spurred a rapidly developing field of artificial intelligence research focused on climate change applications. The climate change AI (CCAI) community works on a diverse, challenging set of problems which often involve physics-constrained ML or heterogeneous spatiotemporal data. It would be desirable to use automated machine learning (AutoML) techniques to automatically find high-performing architectures and hyperparameters for a given dataset. In this work, we benchmark popular AutoML libraries on three high-leverage CCAI applications: climate modeling, wind power forecasting, and catalyst discovery. We find that out-of-the-box AutoML libraries currently fail to meaningfully surpass the performance of human-designed CCAI models. However, we also identify a few key weaknesses, which stem from the fact that most AutoML techniques are tailored to computer vision and NLP applications. For example, while dozens of search spaces have been designed for image and language data, none have been designed for spatiotemporal data. Addressing these key weaknesses can lead to the discovery of novel architectures that yield substantial performance gains across numerous CCAI applications. Therefore, we present a call to action to the AutoML community, since there are a number of concrete, promising directions for future work in the space of AutoML for CCAI. We release our code and a list of resources at https://github.com/climate-change-automl/climate-change-automl.
BibTeX
@misc{tu2022automl,
  author = {Renbo Tu and Nicholas Roberts and Vishak Prasad and Sibasis Nayak and Paarth Jain and Frederic Sala and Ganesh Ramakrishnan and Ameet Talwalkar and Willie Neiswanger and Colin White},
  title = {{AutoML for Climate Change: A Call to Action}},
  year = {2022},
  eprint = {2210.03324},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.03324},
}
Figure