Abstract
Molecular foundation models are rapidly advancing scientific discovery, but their unreliability on out-of-distribution (OOD) samples severely limits their application in high-stakes domains such as drug discovery and protein design. A critical failure mode is chemical hallucination, where models make high-confidence yet entirely incorrect predictions for unknown molecules. To address this challenge, we introduce Molecular Preference-Aligned Instance Ranking (Mole-PAIR), a simple, plug-and-play module that can be flexibly integrated with existing foundation models to improve their reliability on OOD data through cost-effective post-training. Specifically, our method formulates the OOD detection problem as a preference optimization over the estimated OOD affinity between in-distribution (ID) and OOD samples, achieving this goal through a pairwise learning objective. We show that this objective essentially optimizes AUROC, which measures how consistently ID and OOD samples are ranked by the model. Extensive experiments across five real-world molecular datasets demonstrate that our approach significantly improves the OOD detection capabilities of existing molecular foundation models, achieving up to 45.8%, 43.9%, and 24.3% improvements in AUROC under distribution shifts of size, scaffold, and assay, respectively.
Introduction
While artificial intelligence is transforming drug discovery, its deployment in industrial pipelines is often hindered by a lack of reliability. Molecular foundation models (MFMs) frequently exhibit "chemical hallucination," assigning high confidence scores to incorrect predictions when encountering out-of-distribution (OOD) molecules—those significantly different from the training data.
A fundamental issue identified in this work is the "objective-metric misalignment" in current OOD detection methods. Most approaches utilize pointwise objectives, such as Mean Squared Error (MSE) or Binary Cross Entropy (BCE), to estimate a sample's "OOD-ness." However, OOD detection is fundamentally a ranking problem: the goal is to consistently rank in-distribution (ID) samples as "safer" or "more known" than OOD ones. As illustrated in Figure 1, pointwise methods often result in heavily overlapping score distributions, making it difficult to establish a reliable detection threshold.
Methodology: Mole-PAIR
To resolve this misalignment, we propose Molecular Preference-Aligned Instance Ranking (Mole-PAIR). This framework is designed as a lightweight, plug-and-play module compatible with any existing molecular foundation model.
Architecture and Training
Mole-PAIR maintains the efficiency of the underlying foundation model by keeping the large "encoder" frozen. A lightweight Multi-Layer Perceptron (MLP) scoring head is trained on top of these frozen embeddings (Figure 2).
The core innovation lies in the use of Preference Optimization. Rather than evaluating molecules in isolation, Mole-PAIR processes pairs consisting of one ID molecule and one OOD molecule. By employing a pairwise logistic loss—similar to Direct Preference Optimization (DPO) used in Large Language Models—the model is trained to ensure that ID samples are consistently ranked with lower "OOD affinity" than their OOD counterparts.
Theoretical Foundation
We mathematically demonstrate that this pairwise objective directly optimizes for the Area Under the Receiver Operating Characteristic curve (AUROC). This alignment between the training objective and the evaluation metric ensures superior ranking consistency. Furthermore, the gradient of the Mole-PAIR loss naturally prioritizes "hard" pairs—those near the decision boundary or currently misranked—creating a self-paced learning dynamic that focuses on the most critical regions of the score distribution.
Experimental Evaluation
We evaluated Mole-PAIR across five real-world molecular datasets featuring diverse distribution shifts, including DrugOOD (Assay, Scaffold, and Size shifts) and the GOOD Benchmark (GOOD-HIV, GOOD-PCBA, and GOOD-ZINC).
Key Results
Mole-PAIR consistently outperformed standard baselines, including Maximum Softmax Probability (MSP), ODIN, and Mahalanobis distance, across all tested foundation models (e.g., MiniMol, Uni-Mol).
- Significant AUROC Improvements: The method achieved gains of up to 45.8% on ZINC-Size, 43.9% on PCBA-Scaffold, and 24.3% on EC50-Assay.
- Reduced False Positives: In size-based shifts, Mole-PAIR reduced the FPR95 (False Positive Rate at 95% True Positive Rate) to nearly zero, effectively flagging almost all OOD samples without misidentifying known ones.
- Flexibility: Notably, Mole-PAIR does not require property labels during its post-training phase, allowing for broad application across various downstream tasks.
Conclusion
Mole-PAIR addresses the critical challenge of chemical hallucination in molecular foundation models by shifting the focus from pointwise regression to pairwise ranking. By aligning the training objective with the AUROC metric, we achieve significantly more reliable OOD detection. Future work includes exploring unsupervised variants where OOD samples are synthesized by the model and integrating Mole-PAIR into active learning loops to better prioritize experimental synthesis.
Summary updated