Low rank adaptation of chemical foundation models generates effective odorant representations

Grant D. McConachie, Emily Duniec, Florence V. Guerina, Meg A. Younger, Brian DePasqualebioRxiv 2025 · DOI

TL;DR

This paper tackles the challenge of representing odorants for predicting their interactions with olfactory receptors. They found that standard pre-trained chemical foundation models don't outperform traditional hand-crafted features. However, by fine-tuning these models using Low-Rank Adaptation (LoRA), their proposed LORAX model creates more effective odorant representations that better align with neural data and improve prediction accuracy.

What Problem Does This Paper Address?

Predicting how odorants interact with olfactory receptors is a fundamental problem in olfaction research. This is challenging because structurally similar odorants can trigger very different responses in the olfactory system. While researchers have developed various methods to represent odorants (features), there's no single, universally accepted approach. This paper investigates whether modern deep learning techniques, specifically pre-trained "foundation models" from chemistry, can provide better odorant representations than traditional physicochemical descriptors. They found that simply using these pre-trained models as fixed feature extractors doesn't yield significant improvements, suggesting a need for more specialized adaptation.

Methods

The authors explored several approaches:

  • Feature-Based Methods: They evaluated various pre-trained chemical foundation models (e.g., transformer-based like ChemBERTa, GPT, and GNN-based models) and traditional physicochemical descriptors (e.g., MordredDescriptors, ECFP) as fixed feature extractors. These features were then fed into downstream models (Molecule Only - MO, Molecule + Protein - MP, and ProSmith) to predict odorant-receptor affinity.
  • LORAX (LoRA-based Odorant-Receptor Affinity prediction with CROSS-attention): This is their novel approach. LORAX uses a multi-modal transformer architecture that incorporates both chemical and protein foundation models. Crucially, it employs Low-Rank Adaptation (LoRA) to fine-tune the chemical foundation model's representations specifically for olfaction tasks. This fine-tuning process aims to create more task-specific and informative odorant representations. The model then uses an ensemble of XGBoost models for final predictions.

Datasets

The study utilized several publicly available datasets for training and evaluation:

  • Hallem Dataset: Responses of Drosophila melanogaster receptors to odorants.
  • Carey Dataset: Responses of Anopheles gambiae receptors to odorants.
  • M2OR Dataset: A large, curated dataset of mammalian odorant-receptor interactions from various sources. They specifically used a subset (M2OR EC50) for higher quality analysis.

Key Insights and Ideas

  • Foundation Models Aren't a Silver Bullet (Out-of-the-Box): Simply using pre-trained chemical foundation models as fixed feature extractors doesn't significantly outperform well-established physicochemical descriptors for odorant-receptor binding prediction. This suggests that general chemical knowledge isn't directly transferable to the specific nuances of olfaction without adaptation.
  • Fine-Tuning is Crucial: The key to unlocking the potential of foundation models for olfaction lies in fine-tuning. Their proposed LORAX model, which uses LoRA for targeted fine-tuning of chemical representations, demonstrated improved performance.
  • Redundancy in Existing Representations: Analysis revealed significant overlap in information content between many existing chemical representations, both traditional and foundation model-based. This redundancy explains why simply swapping one representation for another often yields similar results.
  • LORAX Creates a More "Olfactory-Like" Space: The LORAX model's learned odorant representations were found to be more dissimilar to physicochemical descriptors and more aligned with neural representations of odorant responses, suggesting it captures more relevant olfactory information.

Key Results

  • Feature-Based Models: In feature-based evaluations, traditional physicochemical descriptors and pre-trained foundation models performed similarly, with neither consistently outperforming the other.
  • LORAX Performance: LORAX achieved comparable or superior performance to existing models like ProSmith on various datasets and generalization tasks. For instance, on the Carey dataset, LORAX showed better generalization to unseen odorants.
  • M2OR Dataset: On the larger M2OR dataset, LORAX demonstrated superior performance compared to other state-of-the-art models, particularly in metrics like MCC and F-score, which are more robust to class imbalance.
  • Neural Alignment: LORAX's odor space was found to be more aligned with underlying neural representations than the original foundation model's space.

Limitations and Future Work

  • Generalization Across Datasets: The authors acknowledge that a key question for future work is how robust LORAX-learned representations are when transferring to different olfactory datasets.
  • Component Contributions: Identifying which specific components of LORAX contribute most to its prediction performance remains an open question.
  • Simultaneous Fine-tuning: They observed that fine-tuning both protein and chemical foundation models simultaneously (LORAX (P+C)) did not improve performance over fine-tuning only the chemical model. Understanding why this is the case is a future research direction.
  • Expanding Foundation Models: The field of chemical foundation modeling is rapidly evolving. Future work could involve systematically benchmarking newer and larger models with LORAX.
  • Perception vs. Binding: While this work focuses on odorant-receptor binding, future research could explore how LORAX-based fine-tuning impacts odor perception prediction, building on prior work in this area.

Summary updated

Abstract

A bstract Featurizing odorants to enable robust prediction of their properties is difficult due to the complex activation patterns that odorants evoke in the olfactory system. Structurally similar odorants can elicit distinct activation patterns in both the sensory periphery (i.e., at the receptor level) and downstream brain circuits (i.e., at a perceptual level). Despite efforts to design or discover features for odorants to better predict how they activate the olfactory system, we lack a universally accepted way to featurize odorants. In this work, we demonstrate that feature-based approaches that rely on pre-trained foundation models do not significantly out-perform classical hand-designed features on odorant-receptor binding tasks, but that targeted foundation model fine-turning can increase model performance beyond these limits. To show this, we introduce a new model that creates olfaction-specific representations: L oRA-based O dorant- R eceptor A ffinity prediction with CROSS -attention ( LORAX ). We compare existing chemical foundation model representations to hand-designed physicochemical descriptors using feature-based methods and identify large information overlap between these representations, highlighting the necessity of fine-tuning to generate novel and superior odorant representations. We show that LORAX produces a feature space more closely aligned with olfactory neural representation, enabling it to outperform existing models on predictive tasks.
BibTeX
@misc{mcconachie2025low,
  author = {Grant D. McConachie and Emily Duniec and Florence V. Guerina and Meg A. Younger and Brian DePasquale},
  title = {{Low rank adaptation of chemical foundation models generates effective odorant representations}},
  year = {2025},
  doi = {10.1101/2025.11.04.686628},
  howpublished = {bioRxiv},
}
Figure