Foundation Models of Scientific Knowledge for Chemistry: Opportunities, Challenges and Lessons Learned

Sameera Horawalavithana, Ellyn Ayton, Shivam Sharma, Scott Howland, Megha Subramanian, Scott Vasquez, Robin Cosbey, Maria Glenski, Svitlana VolkovaACL 2022

TL;DR

This paper introduces the first large-scale foundation models specifically trained on chemistry literature, demonstrating their potential to augment scientific discovery. The authors explore the impact of model size, data diversity, and temporal ordering on performance, finding that diverse data and models trained from scratch outperform general-purpose models.

What Problem Does This Paper Address?

The field of Artificial Intelligence has seen a revolution with the advent of "foundation models" – massive neural networks pre-trained on vast amounts of text data. These models, like GPT-3, have shown remarkable capabilities in various domains. However, their application to specialized scientific fields, such as chemistry, has been limited. This paper addresses the gap in developing and evaluating foundation models tailored for scientific knowledge, specifically focusing on chemistry. The authors aim to understand the opportunities and challenges of applying these powerful models to scientific tasks, enabling chemists to perceive and reason about complex information at an unprecedented scale.

Methods

The core of this work involves pre-training large-scale autoregressive language models using a GPT-2 architecture. The authors experimented with four different model sizes (small, medium, large, and extra-large) to investigate the impact of scaling. A key methodological choice was to pre-train these models from scratch on a chemistry-specific corpus, rather than fine-tuning existing general-purpose models. This approach allows for the development of a specialized vocabulary and a deeper understanding of chemical concepts. The training process leveraged the GPT-NeoX library and was optimized for autoregressive log-likelihood using Adam optimizer and ZeRO optimizer for memory efficiency.

Datasets

The authors curated a substantial dataset of 0.67TB, comprising over 53 million chemistry-focused scientific articles and abstracts. This corpus was collected from 10 diverse sources, including arXiv, Aminer, CORE, MAG, OSTI, CORD-19, PubMed, S2ORC, and Web of Science. To ensure data quality, a rigorous deduplication process was implemented, reducing the dataset size and removing redundant scientific articles. For evaluation, the models were tested on 15+ in-domain chemistry tasks and several standard out-of-domain benchmarks.

Key Insights and Ideas

  • Chemistry-Specific Models Shine: Foundation models pre-trained from scratch on chemistry literature outperform general-purpose models (like GPT-2) on in-domain chemistry tasks. This highlights the value of domain-specific pre-training.
  • Data Diversity Trumps Quantity: The quality and diversity of the training data (e.g., using full-text articles vs. just abstracts) had a more significant impact on model performance than simply increasing the amount of data.
  • Temporal Order Matters (Sometimes): While not universally beneficial, the temporal ordering of documents during pre-training improved performance on specific tasks like SciQ, suggesting that understanding the evolution of scientific knowledge can be advantageous.
  • Model Size is Crucial: Larger models generally led to better downstream performance, aligning with established scaling laws in NLP.
  • Abstracts vs. Full Text: Models trained on full-text articles generally performed better on out-of-domain tasks, suggesting that the richer context in full texts aids generalization. However, for some in-domain tasks, abstract-only models performed comparably or even better, possibly due to the focused nature of abstracts.

Key Results

The paper presents several key quantitative findings:

  • The largest model trained achieved 0.84 accuracy on the SciQ benchmark, outperforming baseline GPT-2 models.
  • Models trained on combined abstract and full-text data achieved a 45% performance advantage on the Pile benchmark compared to full-text only models.
  • The XL (4x) model, trained with a larger batch size, achieved the lowest Lambda and WikiText perplexity values, demonstrating the importance of training scale and batch size.
  • In-domain evaluation showed that the models outperformed baseline GPT-2 on chemistry-specific tasks like SciQ and ARC-E.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Catastrophic Forgetting: Continual pre-training of general-domain models showed a significant performance drop on general language modeling tasks, indicating potential catastrophic forgetting.
  • Vocabulary Mismatch: Fine-tuned models using the original GPT-2 vocabulary struggled with chemistry-specific terms due to fragmented tokenization.
  • Generalization to New Tasks: While models perform well on established benchmarks, their ability to generalize to entirely novel, unseen scientific tasks remains an area for further exploration.
  • Interpretability: Understanding why these models perform well on specific tasks and how they represent chemical knowledge is a crucial next step.
  • Broader Scientific Domains: Extending this work to other scientific disciplines beyond chemistry is a natural progression.

Summary updated

BibTeX
@inproceedings{horawalavithana2022foundation,
  author = {Sameera Horawalavithana and Ellyn Ayton and Shivam Sharma and Scott Howland and Megha Subramanian and Scott Vasquez and Robin Cosbey and Maria Glenski and Svitlana Volkova},
  title = {{Foundation Models of Scientific Knowledge for Chemistry: Opportunities, Challenges and Lessons Learned}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure