Co-training for Low Resource Scientific Natural Language Inference

Mobashir Sadat, Cornelia Caragea

Abstract

Scientific Natural Language Inference (NLI) is often hampered by the high cost of human annotation and the noise inherent in distantly supervised datasets. We present Weighted Co-training (WCT), a method that utilizes training dynamics—specifically confidence and variability—to assign importance weights to data points. Unlike traditional semi-supervised learning that filters out low-confidence examples, WCT retains all data while minimizing the impact of mislabeled instances. Our results demonstrate a 1.5% improvement in Macro F1 on the SciNLI dataset and significant gains in out-of-domain robustness.

Introduction

Scientific NLI involves determining the logical relationship between sentence pairs in research literature (e.g., entailment, contradiction, or neutral). While "distant supervision"—using transition phrases like "therefore" to automatically label data—provides a large-scale alternative to expensive human labeling, it introduces significant noise. Neural models often memorize these incorrect labels, leading to degraded performance. This work addresses the challenge of utilizing noisy data effectively without the data loss associated with traditional filtering.

Weighted Co-training (WCT)

The proposed WCT method trains two classifiers simultaneously, allowing them to exchange beliefs about data trustworthiness. A central component of this approach is Dataset Cartography, which categorizes examples based on their training dynamics across multiple epochs rather than a single snapshot of confidence.

Figure 1: SciNLI-RoBERTa Data Map illustrating the categorization of data into easy-to-learn, hard-to-learn, and ambiguous regions based on confidence and variability.

As shown in Figure 1, data points are divided into three categories:

  • Easy-to-learn: High confidence and low variability; these are likely correctly labeled.
  • Hard-to-learn: Low confidence and low variability; these are likely mislabeled or "noisy."
  • Ambiguous: Moderate confidence and high variability; the model is uncertain about these points.

WCT assigns weights ($\lambda$) to these examples. Crucially, for "ambiguous" examples, the two models receive different weights, forcing them to learn complementary information and preventing premature convergence on biased views.

Experimental Setup

The study utilizes several key datasets:

  • SciNLI: The primary dataset containing ~101k distantly supervised examples.
  • Human-Annotated SciNLI: A newly released gold-standard set of 2,000 curated examples for fine-tuning and evaluation.
  • MSCINLI: Used for out-of-domain (OOD) evaluation across different scientific fields.

Results and Discussion

WCT-CV consistently outperforms standard baselines and existing semi-supervised learning (SSL) techniques.

Figure 2: Performance comparison of various methods on SciBERT and RoBERTa architectures. WCT-CV achieves the highest Macro F1 and Accuracy.

As detailed in Figure 2, the WCT-CV model achieved a 1.5% improvement in Macro F1 over the standard distant supervision baseline. It also surpassed popular SSL methods such as FixMatch and Co-teaching. Notably, the method showed a 1.92% improvement on the out-of-domain MSCINLI dataset, suggesting that the model learns robust logical patterns rather than just memorizing noise.

Comparisons with Large Language Models (LLMs) like Llama-2-13b further highlight the effectiveness of this specialized approach; the LLM achieved only ~52% Macro F1 in a few-shot setting, significantly lower than the ~79% reached by the co-trained models.

Conclusion

Weighted Co-training offers a robust framework for scientific NLI by intelligently weighting noisy data based on training history. By prioritizing "productive disagreement" on ambiguous data and down-weighting likely errors, WCT maximizes the utility of large, distantly supervised datasets. Future work will explore applying this weighting strategy to other low-resource domains such as medical and legal text.

Summary updated

Abstract

Scientific Natural Language Inference (NLI) is the task of predicting the semantic relation between a pair of sentences extracted from research articles. The automatic annotation method based on distant supervision for the training set of SciNLI (Sadat and Caragea, 2022b), the first and most popular dataset for this task, results in label noise which inevitably degenerates the performance of classifiers. In this paper, we propose a novel co-training method that assigns weights based on the training dynamics of the classifiers to the distantly supervised labels, reflective of the manner they are used in the subsequent training epochs. That is, unlike the existing semi-supervised learning (SSL) approaches, we consider the historical behavior of the classifiers to evaluate the quality of the automatically annotated labels. Furthermore, by assigning importance weights instead of filtering out examples based on an arbitrary threshold on the predicted confidence, we maximize the usage of automatically labeled data, while ensuring that the noisy labels have a minimal impact on model training. The proposed method obtains an improvement of 1.5% in Macro F1 over the distant supervision baseline, and substantial improvements over several other strong SSL baselines. We make our code and data available on Github.
BibTeX
@inproceedings{sadat2024co,
  author = {Mobashir Sadat and Cornelia Caragea},
  title = {{Co-training for Low Resource Scientific Natural Language Inference}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.139},
}
Figure