A Systematic Comparison Study on Hyperparameter Optimisation of Graph Neural Networks for Molecular Property Prediction

Yingfang Yuan, Wenjun Wang, Wei Pang

TL;DR

This paper provides a systematic evaluation of how different hyperparameter optimization (HPO) techniques—Random Search, TPE, and CMA-ES—perform when training Graph Neural Networks (GNNs) for molecular property prediction. The authors find that while Random Search is a surprisingly resilient baseline for tiny datasets, Tree-structured Parzen Estimator (TPE) is generally the most efficient and stable choice for researchers working with limited time and computational budgets.

What Problem Does This Paper Address?

Graph Neural Networks have become a go-to tool for predicting molecular properties (like solubility or toxicity), which is essential for drug discovery and materials science. However, GNNs are notoriously sensitive to hyperparameters—settings like learning rate or layer size that aren't "learned" during training but must be set beforehand.

The challenge is twofold:

  1. Computational Cost: Training GNNs is expensive. Finding the "perfect" settings through trial and error is often impossible for researchers without massive GPU clusters.
  2. Small Data: Unlike the massive datasets found in computer vision, molecular datasets are often quite small (hundreds to a few thousand samples). Most HPO methods were designed for "big data" scenarios, and it wasn't clear which ones actually work best in the specialized, data-constrained world of chemistry.

Methods

The researchers compared three distinct HPO strategies:

  • Random Search (RS): The baseline. It picks values at random from a defined range. It’s simple and easy to parallelize but doesn't "learn" from previous failures.
  • Tree-structured Parzen Estimator (TPE): A Bayesian approach. It builds a probabilistic model (a "surrogate") of the objective function. It uses past results to predict which new hyperparameters are most likely to improve performance, balancing exploration of new areas with exploitation of known "good" regions.
  • CMA-ES (Covariance Matrix Adaptation Evolution Strategy): An evolutionary algorithm. it treats HPO like biological evolution, maintaining a "population" of configurations and adapting a multivariate normal distribution to move the population toward better-performing regions of the search space.

The study applied these to a Graph Convolutional Network (GC), optimizing four key parameters: batch size, learning rate, and the sizes of both the graph convolution and fully-connected layers.

Datasets

The authors used three representative benchmarks from MoleculeNet:

  • ESOL: 1,128 records (predicting water solubility).
  • FreeSolv: 642 records (predicting hydration free energy). This is the smallest and most "complex" dataset in the study.
  • Lipophilicity: 4,200 records (predicting octanol/water distribution coefficients). This is the largest dataset used.

Key Insights and Ideas

  • The "Small Data" Paradox: On the tiny FreeSolv dataset, Random Search actually outperformed the more sophisticated TPE and CMA-ES in some tests. The authors suggest that when a dataset is very small and the search space is constrained, sophisticated models can get "stuck" in local optima, whereas the pure randomness of RS helps it stumble upon better solutions.
  • Efficiency is King: In a "one-hour runtime" challenge, TPE was the clear winner. It found better configurations faster than the others, proving that its model-based approach is highly effective when you can't afford hundreds of trials.
  • CMA-ES Needs Room to Run: CMA-ES struggled in several tests. The authors noted that evolutionary strategies typically require a larger number of trials and a larger search space to truly shine; with only 100 trials, it couldn't fully adapt its "population" to the problem.
  • Stability: TPE showed the lowest standard deviation in results, meaning it’s not just fast, but also the most reliable method for producing consistent performance across different runs.

Key Results

  • Lipophilicity (Largest Dataset): TPE demonstrated significantly better performance (lower RMSE) than both RS and CMA-ES.
  • Time Constraints: In the 1-hour test on the ESOL dataset, TPE achieved a Mean RMSE of 0.8634, beating RS (0.8994) and CMA-ES (0.9688), despite RS completing nearly double the number of trials (96 vs 54).
  • Search Space Expansion: When the search space was increased (from $2^{13}$ to $2^{22}$ possible configurations), the performance of TPE and CMA-ES improved significantly compared to their performance in smaller spaces, while RS lagged behind.

Limitations and Future Work

The authors acknowledge a few areas for further exploration:

  • Meta-Hyperparameters: Algorithms like CMA-ES have their own internal settings (like population size). The study used default values, but "tuning the tuner" might improve CMA-ES performance.
  • GNN Variety: The study focused on Graph Convolutional Networks (GC). Future work could test if these HPO trends hold for other architectures like MPNN or Weave.
  • Computational Scale: The study intentionally used a single GPU to reflect the resources of a typical academic lab. Testing these methods on massive parallel systems might change the ranking, particularly for CMA-ES.

Summary updated

Abstract

Graph neural networks (GNNs) have been proposed for a wide range of graph-related learning tasks. In particular, in recent years, an increasing number of GNN systems were applied to predict molecular properties. However, a direct impediment is to select appropriate hyperparameters to achieve satisfactory performance with lower computational cost. Meanwhile, many molecular datasets are far smaller than many other datasets in typical deep learning applications. Most hyperparameter optimization (HPO) methods have not been explored in terms of their efficiencies on such small datasets in the molecular domain. In this paper, we conducted a theoretical analysis of common and specific features for two state-of-the-art and popular algorithms for HPO: TPE and CMA-ES, and we compared them with random search (RS), which is used as a baseline. Experimental studies are carried out on several benchmarks in MoleculeNet, from different perspectives to investigate the impact of RS, TPE, and CMA-ES on HPO of GNNs for molecular property prediction. In our experiments, we concluded that RS, TPE, and CMA-ES have their individual advantages in tackling different specific molecular problems. Finally, we believe our work will motivate further research on GNN as applied to molecular machine learning problems in chemistry and materials sciences.
BibTeX
@inproceedings{yuan2021systematic,
  author = {Yingfang Yuan and Wenjun Wang and Wei Pang},
  title = {{A Systematic Comparison Study on Hyperparameter Optimisation of Graph Neural Networks for Molecular Property Prediction}},
  booktitle = {GECCO '21: Proceedings of the Genetic and Evolutionary Computation Conference},
  year = {2021},
  url = {https://openreview.net/forum?id=zY3akF2QFX},
}

Loading related content...

Figure