SEED: Semantic Knowledge Transfer for Language Model Adaptation to Materials Science

Yeachan Kim, Jun-Hyung Park, SungHo Kim, Juhyeong Park, Sangyun Kim, SangKeun LeeEMNLP 2024 · DOI

TL;DR

Adapting large language models (LLMs) to specialized domains like materials science is challenging due to scarce data. This paper introduces SEED, a novel method that transfers semantic knowledge from existing materials-specific embeddings (Mat2vec) into LLMs, significantly improving their performance on materials science tasks.

What Problem Does This Paper Address?

Pre-trained language models (PLMs), like BERT, are trained on massive, general-domain text corpora (e.g., Wikipedia). While powerful, their performance often degrades when applied to highly specialized fields such as materials science. This is primarily because materials science has a unique and extensive vocabulary, including complex chemical formulas and jargon, which is not well-represented in general training data. Existing methods for adapting PLMs to new domains, such as vocabulary expansion based on downstream dataset frequency, struggle in materials science. This is due to the scarcity and limited coverage of materials science datasets, which can even lead to performance degradation, as observed with methods like AVocaDo. The core problem is how to effectively imbue PLMs with the rich, specialized knowledge of materials science without requiring prohibitively expensive pre-training on massive domain-specific corpora.

Methods

The proposed method, SEED (Semantic Knowledge Transfer), tackles this problem through a two-stage process:

  1. Training Knowledge Transfer (Bridge Network Training):
    • Vocabulary Alignment: SEED first identifies a shared vocabulary (S) between the PLM's existing vocabulary and the external materials embeddings (Mat2vec). It also identifies unique words (U) present only in Mat2vec.
  2. Bridge Network: A novel "bridge network" is introduced. This network, composed of two-layer feed-forward networks, learns to transform the Mat2vec embeddings into representations compatible with the PLM's embedding space.
  3. Optimization: The bridge network is trained using a combined loss function. This includes a reconstruction loss that minimizes the distance between the transformed Mat2vec embeddings and the corresponding PLM embeddings for shared words. Crucially, it also incorporates a relational knowledge distillation component. This loss considers the semantic relationships between words, ensuring that the transferred knowledge preserves the nuanced connections present in the original materials embeddings.
  4. Transferring Materials Knowledge:
    • Selective Transfer: After the bridge network is trained, SEED transfers the knowledge of unique words (U) from Mat2vec into the PLM. To manage memory constraints and focus on relevant terms, this transfer is done selectively. Words are chosen based on their presence in the downstream task's dataset and by finding semantically similar terms using the materials embeddings.
  5. Embedding Expansion: The transformed embeddings for these selected unique words are then interleaved into the PLM's embedding layer.
  6. Fine-tuning: Finally, the adapted PLM is fine-tuned on the specific downstream materials science task.

Datasets

The paper evaluates SEED on four materials science benchmarks:

  • SOFCSF & SOFCNER: Datasets for materials entity recognition.
  • MatScholar: Another dataset for materials entity recognition.
  • Glass Science: A dataset for paragraph classification (determining if a paragraph relates to glass science).
  • SOFC: A dataset for slot-filling tasks.

The external materials embeddings used are Mat2vec, trained on approximately 4.5 million scientific journals. The PLMs used as backbones are BERT (trained on general English corpora) and SciBERT (trained on a scientific corpus).

Key Insights and Ideas

  • Frequency-based adaptation fails in materials science: The paper strongly demonstrates that methods relying solely on word frequency in scarce downstream datasets (like AVocaDo) can be suboptimal and even detrimental in materials science.
  • Semantic knowledge transfer is key: SEED's success highlights that capturing the semantic relationships and latent knowledge within existing domain-specific embeddings is more effective than just adding frequent terms.
  • Relational knowledge distillation is crucial: The inclusion of a relational loss in training the bridge network significantly improves the quality of transferred embeddings, preserving nuanced semantic connections.
  • Selective vocabulary expansion is necessary: For practical application, transferring only relevant unique words based on the downstream task is essential to manage computational resources.
  • SEED is broadly applicable: The method shows strong performance on both general PLMs (BERT) and domain-specific PLMs (SciBERT), indicating its generalizability.

Key Results

  • SEED consistently outperforms existing vocabulary expansion baselines (AdaLM, AVocaDo, Replace) across all four materials science benchmarks when using BERT as a backbone. For example, on the SOFCNER test set, SEED achieves a Macro-F1 score of 0.594, compared to BERT's baseline of 0.569 and AVocaDo's 0.579.
  • Similar improvements are observed when using SciBERT as the backbone, with SEED achieving the highest scores on most benchmarks, such as 0.886 on MatScholar test set.
  • Ablation studies confirm that both the reconstruction loss and the relation loss are critical for SEED's effectiveness, with the relation loss playing a particularly significant role.
  • Visualizations (t-SNE) show that SEED successfully integrates semantically related words (e.g., "nanocrystals," "nanocrystalline" near "electrode," "crystalline") into the PLM's embedding space.

Limitations and Future Work

  • Memory Overhead: While SEED employs selective transfer, transferring all unique words from Mat2vec to PLMs would still incur significant memory overhead. The paper addresses this by selectively transferring words based on task relevance.
  • Future Work: The authors suggest exploring diverse distance metrics and more complex relational structures for the bridge network. They also mention that the current method focuses on expanding the embedding layer, and further research could explore more sophisticated integration strategies. The paper also notes that SEED's effectiveness is demonstrated on specific materials science tasks, and its performance on other specialized domains could be investigated.

Summary updated

BibTeX
@inproceedings{kim2024seed,
  author = {Yeachan Kim and Jun-Hyung Park and SungHo Kim and Juhyeong Park and Sangyun Kim and SangKeun Lee},
  title = {{SEED: Semantic Knowledge Transfer for Language Model Adaptation to Materials Science}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-industry.31},
}
Figure