NLPeople at L+M-24 Shared Task: An Ensembled Approach for Molecule Captioning from SMILES

Shinnosuke Tanaka, Carol Mak, Flaviu Cipcigan, James Barry, Mohab Elkaref, Movina Moses, Vishnudev Kuruvanthodi, Geeth MelACL 2024

TL;DR

This paper tackles the challenge of automatically generating descriptive captions for molecules represented by SMILES strings. The authors propose an ensembled approach that first predicts a molecule's properties and then uses these predictions to select the best caption from different large language models (LLMs), achieving competitive results on the L+M-24 shared task.

What Problem Does This Paper Address?

The core problem is generating accurate and informative natural language captions that describe the properties of molecules. Molecules are often represented by SMILES strings, a compact text-based format. The L+M-24 shared task specifically focuses on translating these SMILES strings into descriptive captions across various domains of molecular science, including biomedical, human interaction, light and electricity, and agriculture/industry. A key challenge is the diversity in how properties are described across these domains, making a one-size-fits-all approach difficult. Existing methods often struggle to capture the nuances of molecular properties and translate them into human-readable text effectively.

Methods

The authors employ a multi-stage approach:

  1. Property Classification: They first treat molecule property prediction as a multi-label classification problem. SMILES strings are converted into molecular fingerprints (using RDKit) and embeddings (from MolT5-Small). These are then concatenated and fed into a classifier to predict various properties. This step aims to provide a structured understanding of the molecule's characteristics.
  2. LLM Caption Generation: Two LLMs, Multitask Text+Chem T5 and LlaSMolMistral, are fine-tuned for direct molecule captioning using SMILES strings as input. Notably, for LlaSMolMistral, they used LoRA (Low-Rank Adaptation) for efficient fine-tuning.
  3. Ensembling: The crucial innovation lies in their ensembling strategy. Based on the output of the property classification model, they decide which LLM's caption to use. If the property classifier predicts at least one property for a given molecule, the caption from LlaSMolMistral is selected. Otherwise, the caption from Multitask T5 is chosen. This conditional selection leverages the strengths of each model based on the confidence of property prediction.

Datasets

  • Training and Evaluation: The primary dataset used is from the L+M-24 Shared Task, which consists of molecule-caption pairs across four high-impact areas of molecular science.
  • Pre-training Data:
    • MolT5 was pre-trained on the C4 Corpus and SMILES from the ZINC-15 dataset.
  • Multitask Text+Chem T5 was trained on SMILES and caption pairs (like ChEBI-20), reaction-products pairs (Pistachio dataset), and experimental procedures (Vaucher et al., 2019).
  • LlaSMolMistral was trained on the SMolInstruct dataset, which includes a molecular captioning task derived from the ChEBI-20 dataset.

Key Insights and Ideas

  • Property Prediction as a Precursor: The paper highlights that predicting molecular properties first can significantly improve captioning performance. This suggests that a structured understanding of a molecule's attributes is a valuable intermediate step for generating descriptive text.
  • LLM Specialization: Different LLMs exhibit varying strengths. LlaSMolMistral performed better when properties were predicted, while Multitask T5 was stronger when no properties could be confidently predicted. This justifies the ensembling approach.
  • Ensembling Strategy: The conditional ensembling based on property prediction confidence is a novel and effective way to combine the outputs of multiple models, leading to improved overall performance.
  • Challenges in Evaluation Metrics: The authors note that metrics like BLEU can be sensitive to minor variations in word order or punctuation, which might not reflect true semantic accuracy, especially given the diverse ways molecules can be described.

Key Results

  • Dev Set Performance: The ensembled model achieved an overall increase score of 15.21 on the dev set and 12.30 on the evaluation set, ranking fourth in the shared task.
  • Translation Metrics: The ensembled model showed strong performance in translation metrics, achieving an overall increase score of 5.52 on the dev set and 5.68 on the eval set.
  • Property Metrics: The ensembled model recorded an increase score of 18.44 on the dev set and 14.50 on the eval set for property metrics.
  • Property Classification Accuracy: The property classifier achieved a high F1 score of 97.86% on the predictable subset of labels.
  • Conditional Performance: When properties were predicted, LlaSMolMistral outperformed Multitask T5 in most translation metrics. Conversely, when no properties were predicted, Multitask T5 significantly outperformed LlaSMolMistral.

Limitations and Future Work

  • Property Prediction Limitations: The property classifier only trained on labels with over 1,500 positive examples, leaving a significant portion of labels unaddressed. This means some molecules might not have any predicted properties, leading to the use of the Multitask T5 model.
  • Ambiguity in Property Descriptions: The paper acknowledges that some property labels can be chemically incorrect or ambiguous (e.g., "fatty" for organoleptic effects vs. general chemical structure), which can affect classification and subsequent captioning.
  • Metric Sensitivity: The authors point out the limitations of BLEU scores, which can be heavily influenced by minor linguistic variations, potentially masking true performance gains.
  • Future Work: The paper suggests that exploring more robust evaluation metrics and further refining the property prediction to cover a wider range of labels would be beneficial. They also hint at the potential for creating realistic performance bounds for molecule-to-text models to better understand benchmark saturation.

Summary updated

BibTeX
@inproceedings{tanaka2024nlpeople,
  author = {Shinnosuke Tanaka and Carol Mak and Flaviu Cipcigan and James Barry and Mohab Elkaref and Movina Moses and Vishnudev Kuruvanthodi and Geeth Mel},
  title = {{NLPeople at L+M-24 Shared Task: An Ensembled Approach for Molecule Captioning from SMILES}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure