Abstract
Scientific Natural Language Inference (NLI) is often hampered by the high cost of human annotation and the noise inherent in distantly supervised datasets. We present Weighted Co-training (WCT), a method that utilizes training dynamics—specifically confidence and variability—to assign importance weights to data points. Unlike traditional semi-supervised learning that filters out low-confidence examples, WCT retains all data while minimizing the impact of mislabeled instances. Our results demonstrate a 1.5% improvement in Macro F1 on the SciNLI dataset and significant gains in out-of-domain robustness.
Introduction
Scientific NLI involves determining the logical relationship between sentence pairs in research literature (e.g., entailment, contradiction, or neutral). While "distant supervision"—using transition phrases like "therefore" to automatically label data—provides a large-scale alternative to expensive human labeling, it introduces significant noise. Neural models often memorize these incorrect labels, leading to degraded performance. This work addresses the challenge of utilizing noisy data effectively without the data loss associated with traditional filtering.
Weighted Co-training (WCT)
The proposed WCT method trains two classifiers simultaneously, allowing them to exchange beliefs about data trustworthiness. A central component of this approach is Dataset Cartography, which categorizes examples based on their training dynamics across multiple epochs rather than a single snapshot of confidence.
As shown in Figure 1, data points are divided into three categories:
- Easy-to-learn: High confidence and low variability; these are likely correctly labeled.
- Hard-to-learn: Low confidence and low variability; these are likely mislabeled or "noisy."
- Ambiguous: Moderate confidence and high variability; the model is uncertain about these points.
WCT assigns weights ($\lambda$) to these examples. Crucially, for "ambiguous" examples, the two models receive different weights, forcing them to learn complementary information and preventing premature convergence on biased views.
Experimental Setup
The study utilizes several key datasets:
- SciNLI: The primary dataset containing ~101k distantly supervised examples.
- Human-Annotated SciNLI: A newly released gold-standard set of 2,000 curated examples for fine-tuning and evaluation.
- MSCINLI: Used for out-of-domain (OOD) evaluation across different scientific fields.
Results and Discussion
WCT-CV consistently outperforms standard baselines and existing semi-supervised learning (SSL) techniques.
As detailed in Figure 2, the WCT-CV model achieved a 1.5% improvement in Macro F1 over the standard distant supervision baseline. It also surpassed popular SSL methods such as FixMatch and Co-teaching. Notably, the method showed a 1.92% improvement on the out-of-domain MSCINLI dataset, suggesting that the model learns robust logical patterns rather than just memorizing noise.
Comparisons with Large Language Models (LLMs) like Llama-2-13b further highlight the effectiveness of this specialized approach; the LLM achieved only ~52% Macro F1 in a few-shot setting, significantly lower than the ~79% reached by the co-trained models.
Conclusion
Weighted Co-training offers a robust framework for scientific NLI by intelligently weighting noisy data based on training history. By prioritizing "productive disagreement" on ambiguous data and down-weighting likely errors, WCT maximizes the utility of large, distantly supervised datasets. Future work will explore applying this weighting strategy to other low-resource domains such as medical and legal text.
Summary updated