Abstract
Scientific Natural Language Inference (NLI) is a challenging task due to the specialized and dense nature of scientific text. While distant supervision offers a scalable way to create large datasets like SCINLI, it introduces significant label noise. This paper presents Weighted Co-training (WCT), a method that utilizes training dynamics—specifically confidence and variability—to assign importance weights to noisy labels rather than filtering them. Our approach achieves a 1.5% improvement in Macro F1 scores on the SCINLI dataset, outperforming established semi-supervised learning (SSL) frameworks and large language models like Llama-2.
Introduction
Natural Language Inference involves determining the logical relationship (entailment, contradiction, or neutral) between sentence pairs. In scientific domains, creating high-quality labeled data is expensive and requires domain expertise. Distant supervision, which labels pairs based on transition words, is a common alternative but results in "noisy" datasets that can lead models to memorize errors. Existing solutions often filter out low-confidence labels, which discards potentially diverse and useful information. This research proposes a way to retain all data while mitigating noise through a weighted dual-classifier system.
Methodology: Weighted Co-training (WCT)
The core innovation of this work is Weighted Co-training (WCT). Unlike traditional co-training, where models exchange high-confidence pseudo-labels, WCT uses the historical training behavior of two models ($\theta_1$ and $\theta_2$) to exchange importance weights.
Dataset Cartography
The weighting mechanism is built upon Dataset Cartography metrics:
- Confidence: The average probability assigned to a label across training epochs.
- Variability: The degree of fluctuation in that probability.
As shown in Figure 1, data points are categorized into three regions: easy-to-learn (high confidence, low variability), hard-to-learn (low confidence, low variability), and ambiguous (high variability).
The Weighting Strategy
WCT assigns high weights to "easy" examples for both models and low weights to "hard" examples. Crucially, "ambiguous" examples receive contrasting weights between the two models. This strategy forces the classifiers to maintain functional divergence, preventing them from collapsing into a single biased perspective and allowing them to learn complementary features.
Datasets and Evaluation
The study utilizes several key datasets:
- SCINLI: Approximately 101k examples labeled via distant supervision.
- Human-Annotated SCINLI: A new gold-standard set of 2,000 human-validated examples, a major contribution of this work.
- MSCINLI: Used for out-of-domain (OOD) evaluation across multiple scientific disciplines.
- ANLI & Dynasent: Used to prove the method generalizes to adversarial NLI and sentiment analysis.
Experimental Results
The proposed WCT-CV model demonstrates superior performance across multiple architectures. As detailed in Figure 2, WCT achieves a Macro F1 of 78.55% with SciBERT and 79.62% with RoBERTa, significantly exceeding the distant supervision baselines and other SSL methods like FixMatch, FlexMatch, and Co-teaching.
Beyond standard benchmarks, WCT showed a 1.92% improvement in out-of-domain tests on MSCINLI. Notably, the method outperformed Llama-2-13b, which achieved only 52.57% Macro F1 in a few-shot setting, underscoring the continued relevance of specialized, smaller models in niche scientific domains.
Discussion and Conclusion
The success of WCT highlights that weighting data based on training history is more effective than simple confidence-based filtering. By preserving dataset diversity and encouraging productive disagreement between models, WCT effectively navigates the noise inherent in distantly supervised scientific text. While the method requires a small human-annotated "seed" set and increased computational resources for dual-model training, it provides a robust framework for low-resource scientific NLI. Future work will explore the application of these training dynamics to other semi-supervised learning domains.
Summary updated