TL;DR
Molecular foundation models frequently suffer from "chemical hallucination"—making confident yet completely erroneous predictions on unfamiliar compounds. Mole-PAIR (Molecular Preference-Aligned Instance Ranking) provides an efficient, plug-and-play post-training remedy: by reframing out-of-distribution (OOD) detection as a pairwise preference optimization problem, it directly optimizes the ranking metric (AUROC) using a frozen encoder and a lightweight scoring head.
What Problem Does This Paper Address?
Pretrained molecular foundation models show immense promise in property prediction and computational drug design. However, deploying them into real-world laboratory workflows is hazardous because of chemical hallucination: when exposed to molecules outside their training distribution (e.g., novel scaffolds, differing molecular sizes, or alternative experimental assay conditions), models often output high-confidence predictions that are entirely wrong.
Most existing OOD detection techniques suffer from two key flaws:
- Model Coupling & Domain Mismatch: Many are adapted directly from vision architectures or tightly coupled with complex graph generative models.
- Objective–Metric Misalignment: Standard techniques rely on pointwise regression or classification losses (such as Mean Squared Error or Binary Cross-Entropy). These force the network to calibrate absolute confidence scores for individual molecules—an objective that easily collapses under distribution shift and is misaligned with standard ranking metrics like AUROC, which depend purely on the relative order between in-distribution (ID) and OOD samples.
Methods
Mole-PAIR reformulates OOD detection as a pairwise preference optimization task inspired by Direct Preference Optimization (DPO) and the Bradley–Terry preference model:
- Frozen Backbone + Lightweight Head: The pretrained molecular foundation encoder (e.g., 2D GNN-based MiniMol or 3D Transformer-based Uni-Mol) remains frozen. The method trains only a tiny multi-layer perceptron (MLP) scoring head ($512 \to 256 \to 128 \to 1$) that outputs an OOD affinity score $E_\phi(S)$.
-
Preference Learning Objective: Rather than predicting a target label, the scoring head is trained on pairs of ID molecules ($S_{\text{in}}$) and generic auxiliary OOD molecules ($S_{\text{out}}$). The model minimizes a logistic pairwise loss: $$\mathcal{L}_{\text{Mole-PAIR}}(\phi) = -\mathbb{E}\left[\log \sigma\left(\beta \cdot \left[E_\phi(S_{\text{out}}) - E_\phi(S_{\text{in}})\right]\right)\right]$$ where $\beta$ controls the temperature/sharpness of preference margins.
-
Gauge-Fixing Regularization: Because pairwise ranking is shift-invariant ($E_\phi(S) \mapsto E_\phi(S) + c$), the authors add a light $\ell_2$ penalty to anchor the mean scores near zero and prevent gradient saturation.
Datasets
The authors evaluated the method across two major molecular OOD benchmarks:
- DrugOOD: Curated from ChEMBL bioassays for ligand-based affinity prediction across both $\text{EC}_{50}$ and $\text{IC}_{50}$ targets. It evaluates three domain shifts based on biochemical properties: Scaffold, Size, and Assay.
- GOOD (Graph OOD Benchmark): Evaluates graph-level distribution shifts on GOOD-HIV (binary classification), GOOD-PCBA (128-assay multi-target classification), and GOOD-ZINC (property regression binarized by median split) under Scaffold and Size covariate shifts.
Key Insights and Ideas
- Direct AUROC Optimization: The authors prove theoretically that minimizing this pairwise logistic loss directly optimizes AUROC. Over a sufficiently expressive function class, the global minimizer recovers the Bayes-optimal ranking induced by the true posterior $\text{Pr}(\text{ID} \mid S)$.
- Hard-Pair Prioritization (Self-Paced Learning): The gradient weight of a pair scales with $w_\beta(\Delta) = \beta \sigma(-\beta \Delta)$. Misranked ($\Delta < 0$) or borderline ($\Delta \approx 0$) pairs naturally receive the strongest gradient signals, whereas comfortably separated pairs are ignored. Optimization effort dynamically focuses exactly where AUROC is being degraded.
- Generalization to Unseen Shifts: Mole-PAIR does not memorize the auxiliary OOD data. Models trained using scaffold shifts as auxiliary OOD successfully generalize to size-based test shifts (and vice versa), showing that the head learns a robust boundary around the ID manifold rather than a narrow signature of the training outliers.
- Extreme Data Efficiency: Even when training OOD data was reduced to a 1:0.01 ID:OOD ratio (a mere 20 auxiliary OOD molecules), Mole-PAIR still outperformed standard baselines trained on full 1:1 balanced splits.
Key Results
- Substantial AUROC Gains: Compared to standard baselines (MSP, ODIN, Energy, Mahalanobis distance, KNN, LOF), Mole-PAIR achieved average improvements of 28.3% on AUROC, 28.5% on AUPR, and reduced FPR95 by 25.3% across all datasets.
- Near-Perfect Size Shift Detection: On size-based shifts (e.g., MiniMol on $\text{EC}_{50}$-Size and Uni-Mol on $\text{IC}_{50}$-Size), AUROC reached 1.000, reducing false positive rates at 95% true positive rate (FPR95) down to 0.000.
- Tough Scaffold Shifts: On the challenging MiniMol $\text{EC}_{50}$-Scaffold split, Mole-PAIR increased AUROC from 0.677 (MSP) to 0.970, slashing the FPR95 from 0.725 to 0.178.
- Outperforming SOTA Graph Baselines: Mole-PAIR outperformed dedicated, architecture-specific graph OOD methods including GraphDE, GOOD-D, and diffusion-based PGR-MOOD.
- Minimal Compute Overhead: Training the scoring head required negligible overhead (~112 seconds and ~1.60 GB of VRAM), making it practical for standard production pipelines.
Limitations and Future Work
- Challenging Assay Shifts: While Mole-PAIR improved AUROC under assay distribution shifts (reaching ~0.65–0.71), FPR95 remained relatively high (>0.80). Subtle experimental variations across biological assays remain difficult to distinguish from feature embeddings alone.
- Dependence on Negative Samples: Although robust to severe data scarcity (requiring as few as 20 samples), the method still requires access to a generic auxiliary set of non-ID chemical structures during post-training.
- Broader Modalities: Future work includes extending preference-based ranking to multi-modal foundation models (e.g., combining 3D protein pockets with 2D ligand graphs) and exploring active learning workflows where low-affinity compounds trigger automated experimental validation.
Summary updated