Perplexity-Based Molecule Ranking and Bias Estimation of Chemical Language Models

Michael Moret, Francesca Grisoni, Paul Katzberger, Gisbert SchneiderJournal of Chemical Information and Modeling 2022 · OpenReview

TL;DR

This paper introduces perplexity as a novel metric for ranking molecules generated by chemical language models (CLMs), offering a way to assess "design quality" directly from the model's learned probabilities. This method not only helps identify promising candidates but also reveals and quantifies "pretraining bias," a common issue where models rely too heavily on general knowledge rather than specific training objectives.

What Problem Does This Paper Address?

Designing new molecules with desired properties is a major goal in drug discovery and materials science. Chemical Language Models (CLMs) are powerful tools for this, generating novel molecular structures represented as text (like SMILES strings). However, a key challenge is evaluating the quality of these generated molecules before expensive synthesis and testing. Existing methods often rely on external scoring functions or similarity measures, which can be computationally intensive or miss subtle aspects of molecular design. Furthermore, CLMs trained using transfer learning (pretraining on large datasets and fine-tuning on smaller, specific ones) can inadvertently carry over biases from the pretraining phase, leading to molecules that don't truly align with the fine-tuning objectives. This paper aims to address these issues by providing an intrinsic, model-based scoring mechanism.

Methods

The core of this work is the application of perplexity, a metric commonly used in natural language processing, to evaluate chemical language models. Perplexity quantifies how well a language model predicts a sequence of characters. In this context, a lower perplexity score for a generated SMILES string indicates that the string is more probable according to the CLM's learned distribution, suggesting it better matches the patterns learned during training.

The authors compare two common CLM generation strategies:

  • Multinomial Sampling: This method samples the next character based on its probability distribution, allowing for broader exploration of chemical space but potentially generating less focused molecules.
  • Beam Search: This is a more "greedy" approach that explores a limited set of the most probable next characters, leading to more focused but potentially less diverse molecules.

To address pretraining bias, they introduce a "delta score." This score is calculated by comparing the rank of a generated molecule based on its perplexity after fine-tuning versus its rank based on perplexity before fine-tuning (i.e., using only the pretrained model). A negative delta score suggests the molecule is favored by the pretrained model, indicating potential bias.

Datasets

The study utilizes the ChEMBL database (version 28) for its experiments.

  • A large pretraining set of approximately 1.6 million molecules was used.
  • For fine-tuning, 10 specific macromolecular targets were selected. For each target, fine-tuning sets of varying sizes (5, 10, 20, and 40 molecules) were created, all possessing a pChEMBL activity value greater than 6.

Key Insights and Ideas

  • Perplexity as an Intrinsic Quality Score: The paper demonstrates that perplexity can effectively rank generated molecules based on how well they align with the CLM's learned probability distribution. This offers a direct, model-internal way to assess design "goodness."
  • Multinomial Sampling's Advantage: Contrary to expectations, multinomial sampling, when combined with perplexity scoring, often yields better results (lower perplexity) than beam search, especially in low-data regimes. This suggests that its broader exploration of chemical space is beneficial, and perplexity can effectively filter the promising candidates from this diverse pool.
  • Quantifying Pretraining Bias: The introduction of the "delta score" is a significant contribution. It provides a concrete method to identify and quantify how much a CLM's output is influenced by its initial pretraining versus its specific fine-tuning objectives. This is crucial for ensuring generated molecules are relevant to the target task.
  • Perplexity and Delta Score Synergy: Combining perplexity for ranking and the delta score for bias detection offers a more robust approach to molecule prioritization than using either metric alone.

Key Results

  • Perplexity scores showed a moderate correlation (around 0.5) with traditional similarity measures (Morgan and pharmacophore fingerprints), indicating it captures relevant chemical features.
  • Multinomial sampling generally produced molecules with lower perplexity scores compared to beam search, especially when considering the top-scoring molecules.
  • The delta score analysis revealed that a significant portion (10-40%) of molecules generated by CLMs could be attributed to pretraining bias, particularly in the early stages of fine-tuning.
  • When filtering out designs with substructure similarity greater than 50% to the fine-tuning set, multinomial sampling still outperformed beam search in identifying molecules with lower perplexity in 72% of cases for small fine-tuning sets.

Limitations and Future Work

  • Low-Data Regimes: While the study focuses on low-data regimes, the performance of perplexity and the delta score might differ with larger fine-tuning datasets.
  • Temperature Parameter: The study used a fixed temperature (T=1) for multinomial sampling. Future work could explore the interplay between perplexity and varying temperature parameters.
  • SMILES Augmentation: The authors suggest exploring SMILES augmentation as a future direction.
  • Broader Applications: Further experiments are needed to determine the full impact of this approach on molecular de novo designs, including screening large collections of commercially available compounds.

Summary updated

BibTeX
@article{moret2022perplexity,
  author = {Michael Moret and Francesca Grisoni and Paul Katzberger and Gisbert Schneider},
  title = {{Perplexity-Based Molecule Ranking and Bias Estimation of Chemical Language Models}},
  journal = {Journal of Chemical Information and Modeling},
  year = {2022},
  url = {https://openreview.net/forum?id=vom7IuTwpQ},
}
Figure