Abstract
Standard regression techniques in molecular machine learning often suffer from unstable gradients and poor generalization. This paper introduces Distributional Mixture of Experts (DMoE), a novel approach that replaces scalar regression with "histogram regression." By predicting a probability distribution over discrete bins and employing a hybrid loss function, DMoE addresses the inherent flaws of quantization error and distance bias in previous histogram methods. The approach yields significant accuracy improvements across major benchmarks like OC20 and QM9, while providing robust, built-in uncertainty quantification.
Introduction
In molecular property prediction, the goal is typically to predict a continuous scalar value, such as the adsorption energy of a molecule. Most models minimize the distance between a predicted scalar and the ground truth using Mean Squared Error (MSE). However, this can lead to training instabilities. While "soft targets" have improved classification tasks, their regression counterpart—histogram regression—has been hampered by two main issues: quantization error, where discrete bins lose resolution, and distance bias, where standard losses fail to recognize that being "close" in bin space is better than being "far."
Methodology: The DMoE Framework
The authors propose the Distributional Mixture of Experts (DMoE) to overcome these limitations through a multi-faceted architecture and a specialized loss function.
Key Innovations
- Non-linear Binning: Instead of uniform bins, DMoE uses bins distributed according to a Gaussian prior. This concentrates resolution in high-density data regions, reducing average quantization error.
- Multi-Head Shifting: The model employs multiple output heads with slightly offset bin boundaries. Averaging these predictions acts as a form of anti-aliasing, smoothing the discrete output into a more continuous-like representation.
- Hybrid Loss Function: To solve distance bias, the authors combine standard Histogram Loss ($L_{HL}$) with a Distance-based Loss ($L_{DL}$). The $L_{DL}$ term calculates the $L_1$ distance between the expected value of the predicted distribution and the true scalar target: $$L_{DMoE} = \alpha_{HL} L_{HL} + \alpha_{DL} L_{DL}$$
Addressing Distance Bias
A critical contribution of DMoE is the explicit penalization of distance errors. Standard cross-entropy treats all incorrect bins equally, regardless of their distance from the target.
As illustrated in Figure 6, the hybrid loss allows the model to distinguish between "near misses" and "far misses," providing a much stronger training signal than histogram loss alone.
Experimental Results
The DMoE method was evaluated on three major benchmarks: Open Catalyst 2020 (OC20), MD17, and QM9.
- OC20 Performance: Integrating DMoE into the GemNet-dT architecture resulted in an 8.4% relative improvement in energy Mean Absolute Error (MAE). More impressively, it achieved a 34.3% improvement in the "Energy within Threshold" (EwT) metric, demonstrating superior precision.
- QM9: The method improved threshold accuracy by an average of 5% across various quantum chemical properties.
- Uncertainty Quantification: By calculating the entropy or KL-divergence of the predicted distributions, DMoE provides a natural measure of model confidence. This uncertainty metric outperformed standard ensemble methods without requiring additional training passes.
Discussion and Conclusion
DMoE demonstrates that treating regression as a distribution learning problem can significantly stabilize training and improve accuracy. The use of non-linear binning and multi-head shifting effectively mitigates quantization errors, while the hybrid loss resolves the distance bias inherent in discrete classification-style losses.
While highly effective, the method currently faces challenges with out-of-distribution (OOD) data, as bin ranges are fixed during training. Future work exploring dynamic binning could allow the model to adapt its resolution to unseen data ranges. Despite this, DMoE represents a significant step forward in robust and interpretable molecular regression.
Summary updated