Abstract
Extracting structured knowledge from scientific literature is often hindered by the high cost of expert annotation and significant class imbalance. We propose a Semi-supervised Transfer Learning (STL) framework designed for joint entity and relation extraction in low-resource settings. By integrating cross-domain transfer learning with a distribution-based rebalancing strategy for self-training, our approach effectively leverages unlabeled data to outperform standard supervised and self-training baselines.
Introduction
Information Extraction (IE), encompassing Named Entity Recognition (NER) and Relation Extraction (RE), is critical for synthesizing scientific knowledge. However, two primary challenges persist: the Expertise Gap, where annotation requires specialized knowledge, and Class Imbalance, where common relations (e.g., "Used-for") dominate rare but important ones (e.g., "Compare"). While many semi-supervised methods simplify the task by assuming pre-identified entities, this work addresses the more complex challenge of joint extraction from raw text under data scarcity.
Methodology
The STL framework is built upon SpERT, a span-based joint extraction model. It employs three core components to maximize learning from limited labels:
- Transfer Learning: The model's encoder is initialized using the top layers of a model pre-trained on a related scientific domain (e.g., transferring knowledge from computer science to biomedicine). This allows the model to inherit high-level semantic features relevant to scientific discourse.
- Self-Training (SSL): The model generates pseudo-labels for a large pool of unlabeled data. High-confidence predictions are iteratively incorporated into the training set.
- Rebalancing Strategy: To mitigate bias toward majority classes, a distribution-based sampling rate is applied during self-training. This ensures that instances predicted as minority classes are prioritized for inclusion in the training set, even if their overall frequency is low.
Experimental Setup
The framework was evaluated on two benchmarks: SciERC (500 AI abstracts) and ADE (4,722 biomedical sentences). To simulate low-resource conditions, only 20% of the labeled data was utilized for training, with the remainder treated as unlabeled.
Results and Discussion
Experimental results demonstrate that the STL framework consistently outperforms both the supervised baseline and standard self-training (ST) across both datasets.
The Noise Trap and Rebalancing
A key finding is that standard self-training (ST) can sometimes perform worse than simple supervised learning. This occurs because noisy pseudo-labels can lead the model to reinforce its own errors. The STL framework avoids this through its rebalancing strategy, which specifically targets minority class performance. For example, in the SciERC dataset, the "Evaluate-for" relation saw a significant F1 improvement from 4.2 to 12.0 when rebalancing was applied.
Transfer Learning Efficacy
The decision to transfer only the top layers of the SciBERT encoder proved effective. These layers capture domain-agnostic semantic structures necessary for extraction, while lower layers remain more tied to the specific vocabulary of the source domain.
Conclusion
The STL framework provides a robust solution for scientific IE when labeled data is scarce. By combining cross-domain knowledge transfer with an active rebalancing of unlabeled data, it achieves superior performance on both entity and relation extraction. Future work will focus on refining the rebalancing strategy, as some rare classes (e.g., "Part-of") showed inconsistent gains, suggesting a need for more nuanced sampling techniques.
Summary updated