Distribution Learning for Molecular Regression

Nima Shoghi, Pooya Shoghi, Anuroop Sriram, Abhishek DasarXiv 2024 · DOI · arXiv

Abstract

Standard regression techniques in molecular machine learning often suffer from unstable gradients and poor generalization. This paper introduces Distributional Mixture of Experts (DMoE), a novel approach that replaces scalar regression with "histogram regression." By predicting a probability distribution over discrete bins and employing a hybrid loss function, DMoE addresses the inherent flaws of quantization error and distance bias in previous histogram methods. The approach yields significant accuracy improvements across major benchmarks like OC20 and QM9, while providing robust, built-in uncertainty quantification.

Introduction

In molecular property prediction, the goal is typically to predict a continuous scalar value, such as the adsorption energy of a molecule. Most models minimize the distance between a predicted scalar and the ground truth using Mean Squared Error (MSE). However, this can lead to training instabilities. While "soft targets" have improved classification tasks, their regression counterpart—histogram regression—has been hampered by two main issues: quantization error, where discrete bins lose resolution, and distance bias, where standard losses fail to recognize that being "close" in bin space is better than being "far."

Methodology: The DMoE Framework

The authors propose the Distributional Mixture of Experts (DMoE) to overcome these limitations through a multi-faceted architecture and a specialized loss function.

Figure 1: The DMoE architecture overview, showing the transformation of scalar targets into probability distributions and the hybrid loss calculation.

Key Innovations

  1. Non-linear Binning: Instead of uniform bins, DMoE uses bins distributed according to a Gaussian prior. This concentrates resolution in high-density data regions, reducing average quantization error.
  2. Multi-Head Shifting: The model employs multiple output heads with slightly offset bin boundaries. Averaging these predictions acts as a form of anti-aliasing, smoothing the discrete output into a more continuous-like representation.
  3. Hybrid Loss Function: To solve distance bias, the authors combine standard Histogram Loss ($L_{HL}$) with a Distance-based Loss ($L_{DL}$). The $L_{DL}$ term calculates the $L_1$ distance between the expected value of the predicted distribution and the true scalar target: $$L_{DMoE} = \alpha_{HL} L_{HL} + \alpha_{DL} L_{DL}$$

Addressing Distance Bias

A critical contribution of DMoE is the explicit penalization of distance errors. Standard cross-entropy treats all incorrect bins equally, regardless of their distance from the target.

Figure 6: Illustration of distance bias. In (a), two distributions with identical histogram loss (HL) are shown; the distance-based loss (DL) correctly identifies the red distribution as superior because it is closer to the ground truth.

As illustrated in Figure 6, the hybrid loss allows the model to distinguish between "near misses" and "far misses," providing a much stronger training signal than histogram loss alone.

Experimental Results

The DMoE method was evaluated on three major benchmarks: Open Catalyst 2020 (OC20), MD17, and QM9.

  • OC20 Performance: Integrating DMoE into the GemNet-dT architecture resulted in an 8.4% relative improvement in energy Mean Absolute Error (MAE). More impressively, it achieved a 34.3% improvement in the "Energy within Threshold" (EwT) metric, demonstrating superior precision.
  • QM9: The method improved threshold accuracy by an average of 5% across various quantum chemical properties.
  • Uncertainty Quantification: By calculating the entropy or KL-divergence of the predicted distributions, DMoE provides a natural measure of model confidence. This uncertainty metric outperformed standard ensemble methods without requiring additional training passes.

Discussion and Conclusion

DMoE demonstrates that treating regression as a distribution learning problem can significantly stabilize training and improve accuracy. The use of non-linear binning and multi-head shifting effectively mitigates quantization errors, while the hybrid loss resolves the distance bias inherent in discrete classification-style losses.

While highly effective, the method currently faces challenges with out-of-distribution (OOD) data, as bin ranges are fixed during training. Future work exploring dynamic binning could allow the model to adapt its resolution to unseen data ranges. Despite this, DMoE represents a significant step forward in robust and interpretable molecular regression.

Summary updated

Abstract

Using "soft" targets to improve model performance has been shown to be effective in classification settings, but the usage of soft targets for regression is a much less studied topic in machine learning. The existing literature on the usage of soft targets for regression fails to properly assess the method's limitations, and empirical evaluation is quite limited. In this work, we assess the strengths and drawbacks of existing methods when applied to molecular property regression tasks. Our assessment outlines key biases present in existing methods and proposes methods to address them, evaluated through careful ablation studies. We leverage these insights to propose Distributional Mixture of Experts (DMoE): A model-independent, and data-independent method for regression which trains a model to predict probability distributions of its targets. Our proposed loss function combines the cross entropy between predicted and target distributions and the L1 distance between their expected values to produce a loss function that is robust to the outlined biases. We evaluate the performance of DMoE on different molecular property prediction datasets -- Open Catalyst (OC20), MD17, and QM9 -- across different backbone model architectures -- SchNet, GemNet, and Graphormer. Our results demonstrate that the proposed method is a promising alternative to classical regression for molecular property prediction tasks, showing improvements over baselines on all datasets and architectures.
BibTeX
@misc{shoghi2024distribution,
  author = {Nima Shoghi and Pooya Shoghi and Anuroop Sriram and Abhishek Das},
  title = {{Distribution Learning for Molecular Regression}},
  year = {2024},
  eprint = {2407.20475},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.20475},
}
Figure