TL;DR
This paper introduces a novel pretraining method for domain-specific language models that leverages LLM-generated data, either from existing ontologies or automatically extracted keywords. This approach significantly improves performance, especially in low-resource settings, and offers a robust alternative to traditional masked language modeling (MLM) by mitigating issues like model collapse.
What Problem Does This Paper Address?
Transformer encoder models like BERT have achieved remarkable success in various NLP tasks. However, their effectiveness in specialized domains (e.g., scientific research) often requires extensive domain-specific pretraining. This becomes a challenge in fields with limited training data or readily available structured knowledge. Traditional MLM pretraining, while effective, can be data-hungry and may not efficiently inject nuanced domain knowledge. The authors aim to bridge this gap by proposing methods to effectively inject domain knowledge into encoder models using LLM-generated synthetic data, exploring both scenarios where structured ontologies are available and where they are not.
Methods
The core of the paper is a similarity-based pretraining (SIM) strategy. Instead of traditional MLM, which masks tokens, SIM trains encoder models to embed definitions of the same concept close together in the embedding space, while pushing definitions of different concepts further apart. This is achieved using a triplet margin loss, where definitions of the same concept act as anchors and positives, and definitions of different concepts act as negatives.
To enhance this, the authors also incorporate concept relatedness by treating related concepts (identified through ontology links or co-occurrence statistics) as additional positive pairs in the loss function. This encourages related concepts to be embedded closer to each other than unrelated ones.
The paper explores two main data sources for this SIM pretraining:
- Ontology-derived data: Concepts, definitions, and relationships are extracted from existing domain-specific ontologies (INBIO and ENVO for invasion biology). LLMs are used to generate additional definitions for these concepts.
- LLM-extracted data: For domains lacking ontologies, concepts (keywords) are extracted from scientific abstracts using an LLM. Definitions are then generated for these keywords, and relationships are inferred from co-occurrence patterns in the abstracts.
The authors also investigate combining SIM pretraining with traditional MLM pretraining, hypothesizing synergistic effects.
Datasets
The primary domain for evaluation is invasion biology.
- Ontology-derived dataset: Created from the INBIO and ENVO ontologies, containing 5,197 ontology-derived concepts with definitions (original + 5 LLM-generated).
- LLM-extracted dataset: Derived from 5,000 scientific abstracts, yielding 23,597 unique keywords with generated definitions and co-occurrence-based relations.
- Abstract sentences: A collection of 37,786 paper titles and abstracts from the invasion biology field was used for MLM pretraining. Experiments were conducted on subsets of 5,000 abstracts.
A new benchmark was compiled from existing studies for evaluating performance on invasion biology tasks, comprising four distinct tasks:
- INAS Classification: 10-class classification of hypotheses in abstracts.
- INAS Span Prediction: Token-level prediction of hypothesis evidence.
- EICAT Impact Classification: Classification of ecological impact in scientific texts.
- EICAT Impact Evidence: Binary sentence classification for evidence of impact.
Key Insights and Ideas
- Ontologies are valuable, but not strictly necessary: While ontology-derived data leads to slightly better performance, LLM-extracted keywords and definitions can achieve comparable results, especially when combined with abstract sentences. This makes the approach adaptable to domains lacking structured knowledge.
- SIM pretraining is a viable alternative to MLM: SIM pretraining alone, especially on ontology data, achieves performance on par with MLM pretraining on real data, demonstrating its effectiveness in injecting domain knowledge.
- Concept relatedness matters: Incorporating relational information between concepts significantly boosts performance, suggesting that understanding how entities connect is crucial for domain understanding.
- Synergy between SIM and MLM: Combining SIM pretraining (especially with ontology data) and MLM pretraining on abstract sentences yields the best results, outperforming either method alone. This suggests that SIM excels at concept-level understanding, while MLM captures broader language patterns.
- LLM-generated data is robust against model collapse: While synthetic data can lead to model collapse in MLM, SIM pretraining is less susceptible. Combining SIM with MLM effectively mitigates this risk, making it a more stable approach for leveraging synthetic data.
Key Results
- The combined SIM (ontology data) + MLM (abstract sentences) approach achieved the highest benchmark score of 0.538, a significant improvement over MLM on abstract sentences alone (0.507).
- Even with LLM-extracted keywords, the combined SIM + MLM approach on just 5,000 abstracts outperformed established models like SciDeBERTa, which were trained on millions of abstracts.
- SIM pretraining on keyword definitions from 5,000 abstracts outperformed MLM pretraining on the same dataset, validating the LLM's ability to enrich the data.
- An ablation study showed that removing concept relatedness information led to a performance drop from 0.507 to 0.498.
- Statistical significance tests confirmed the superiority of most pretraining methods over the DeBERTa base, the benefits of combining SIM and MLM, and the value of ontology data over LLM-extracted keywords.
Limitations and Future Work
- Domain Generalizability: The study focused on invasion biology. Its applicability to other domains needs further evaluation.
- Ontology Features: The study primarily used concepts, definitions, and links. Untapped potential may lie in other ontology features like relation types, domains, and ranges.
- LLM Data Quality: The correctness and quality of LLM-generated data and extracted concepts were not rigorously assessed. Biases or inaccuracies could be introduced.
- Future Work: The authors suggest exploring other ontology features and conducting more extensive evaluations across diverse domains. They also highlight the potential of their approach for low-resource settings and its robustness against model collapse as significant advancements.
Summary updated