Knowlab’s Submission to L+M Shared Task: All you need is continued pretraining of chemistry texts even for molecule captioning

Yunsoo Kim, Honghan WuACL 2024

TL;DR

This paper explores how to improve molecule captioning (describing molecular structures in natural language) by continuing to pre-train a small language model (Phi-3-mini-4k) on a large corpus of chemistry texts, even without direct exposure to molecular representations like SMILES. The results show a significant performance boost, demonstrating that general chemical knowledge is highly beneficial for this task.

What Problem Does This Paper Address?

The field of natural language processing (NLP) is increasingly intersecting with chemistry, particularly in areas like drug discovery and predicting chemical properties. A key challenge is enabling models to understand and generate descriptions of molecular structures, a task known as molecule captioning. This requires translating between molecular representations, such as SMILES (Simplified Molecular Input Line Entry System), and natural language. While large language models (LLMs) have shown promise, effectively training them for this specialized domain is resource-intensive, often requiring extensive domain-specific pretraining and fine-tuning on large, specialized datasets. This paper addresses the need for more efficient methods to imbue language models with chemical knowledge for molecule captioning, specifically investigating the impact of continued pretraining on general chemistry texts.

Methods

The authors employed a two-stage approach using the Phi-3-mini-4k, a small language model (SLM) with 3.8 billion parameters:

  1. Continued Pretraining: The Phi-3-mini-4k model was further pretrained on a substantial corpus of 90 million chemistry textbooks and abstracts. Crucially, this pretraining corpus did not directly include molecular representations like SMILES, aiming to assess the benefit of general chemical language understanding.
  2. Instruction Tuning: Following pretraining, the model underwent instruction tuning using approximately 150,000 question-answering pairs. These pairs covered both SMILES-based questions and general chemistry knowledge, designed to refine the model's ability to handle specific chemical queries and representations.

For the final molecule captioning task, the model was fine-tuned using a low-rank adaptation (LoRA) technique for efficiency.

Datasets

  • Pretraining Corpus: 90 million tokens comprising chemistry textbooks (sourced from HuggingFace) and an additional 82 million tokens from chemical journal abstracts published by the Royal Society of Chemistry.
  • Instruction Tuning Datasets:
    • SMILES Question Answering Pairs: Approximately 129,699 pairs from the ChEMBL dataset, focusing on questions related to chemical properties derived from SMILES representations.
  • CAMEL-AI Chemistry Dataset: A dataset generated using GPT-4, containing 20,000 instruction pairs across 25 topics and 25 subtopics, designed to cover a broad range of chemical queries.

Key Insights and Ideas

  • Pretraining without SMILES is surprisingly effective: The most striking finding is that continued pretraining on general chemistry texts alone, without direct exposure to molecular representations like SMILES, significantly boosted performance in molecule captioning. This suggests that a strong foundation in chemical language and concepts is a critical prerequisite.
  • Small models can be powerful with domain adaptation: By leveraging a relatively small model (Phi-3-mini-4k) and applying targeted pretraining and instruction tuning, the authors achieved competitive results, even outperforming a larger baseline model (MolT5-base).
  • Domain-specific knowledge is key: The results underscore the importance of infusing language models with specialized knowledge for tasks in scientific domains. General language models, while capable, struggle with the nuances of chemistry-specific language and representations.
  • Instruction tuning refines capabilities: While pretraining provides a broad understanding, instruction tuning further hones the model's ability to handle specific tasks and query formats, like SMILES-based questions.

Key Results

The paper reports significant improvements in molecule captioning metrics after continued pretraining and instruction tuning:

  • Continued Pretraining (PT) alone: Led to an almost 300% increase in BLEU scores, a 67% increase in ROUGE scores, and a 19% increase in Meteor scores compared to the baseline Phi-3 model.
  • Continued Pretraining + Instruction Tuning (PT+IT): Further improved performance across all metrics.
  • Final Model (Phi-3-KnowChem): Outperformed the MolT5-base model on the L+M-24 shared task leaderboard.
  • Qualitative Analysis: The baseline Phi-3 model produced repetitive and grammatically flawed captions. The trained models, especially with beam search decoding, generated more coherent and relevant descriptions.

Limitations and Future Work

  • Computational Constraints: The authors acknowledge that computational limitations restricted the size and complexity of the models they could train. They suggest that larger versions of the Phi-3 model (7B, 14B) could potentially yield even better performance.
  • Limited Rigorous Testing: The Phi-3-KnowChem model has not been rigorously tested for real-world chemical applications or decision-making processes, and thus may not be suitable for such uses.
  • Future Directions: The authors suggest further in-depth analysis to fully understand model performance nuances in the chemical domain. They also propose exploring other tasks, such as chemical property prediction, to provide more comprehensive assessments of LM capabilities in chemistry.

Summary updated

BibTeX
@inproceedings{kim2024knowlabs,
  author = {Yunsoo Kim and Honghan Wu},
  title = {{Knowlab’s Submission to L+M Shared Task: All you need is continued pretraining of chemistry texts even for molecule captioning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure