Addressing Limited Data for Textual Entailment Across Domains

Chaitanya P. Shivade, Preethi Raghavan, Siddharth PatwardhanACL 2016 · DOI

Abstract

Textual Entailment (TE) is a fundamental NLP task that determines if a hypothesis logically follows from a premise. While successful in general domains, TE faces significant hurdles in specialized fields like medicine due to domain shifts and the high cost of expert annotation. This paper introduces a clinical dataset and evaluates strategies—specifically self-training and active learning—to mitigate data scarcity. Our findings show that these methods can improve F-scores by up to 15% using unlabeled data or achieve baseline performance using less than 7% of the total labeled dataset.

Introduction: The Challenge of Specialized Domains

Textual Entailment (TE) systems are typically trained on newswire data, which does not translate well to the medical domain. Specialized fields present two primary challenges: a significant domain gap in terminology and structure, and a severe scarcity of labeled data. Annotating medical records requires clinical expertise and is often restricted by privacy regulations.

Example of clinical text from Electronic Medical Records (EMRs) showing domain-specific structure and jargon.

The authors focus on entailment search, which involves identifying all sentences in a corpus that support a specific clinical hypothesis, such as "Does this patient have a history of hypertension?"

Methodology

The research employs a three-pronged approach to bridge the data gap:

  1. Supervised Baseline (ENT): A logistic regression classifier utilizing lexical, syntactic, and domain-specific features. Key to its success is the integration of the Unified Medical Language System (UMLS) to handle medical concepts and a specialized translator for mapping jargon to common terms.
  2. Self-Training: This semi-supervised method leverages unlabeled data. The model iteratively labels unlabeled pairs; those exceeding a confidence threshold ($\tau$) are added to the training set for subsequent rounds.
  3. Active Learning: Using uncertainty sampling, the system identifies instances where the classifier is most uncertain (probabilities near 0.5). Human experts then label only these high-value instances, maximizing the information gain per label.

Data and Experimental Setup

The study compares two domains:

  • Newswire: Standard datasets from PASCAL RTE challenges.
  • Clinical: A novel dataset derived from Electronic Medical Records (EMRs), where medical students annotated the relationship between clinical hypotheses and patient notes.
  • Unlabeled Resources: Large-scale unlabeled corpora were used, including ~43k newswire pairs and ~623k clinical pairs.

Results and Discussion

The experimental results highlight the effectiveness of leveraging unlabeled data and strategic sampling.

The Impact of Self-Training

Self-training provided substantial performance boosts. The F-score increased by 15% on newswire (0.39 to 0.54) and 13% on clinical data (0.23 to 0.36). A critical finding was the sensitivity of the system to the confidence threshold ($\tau$).

Performance metrics (Precision, Recall, and F1-Score) for clinical and newswire data across varying confidence thresholds in self-training.

As shown in the figures above, lower thresholds tend to increase recall but introduce noise, while higher thresholds maintain precision at the cost of diversity.

Efficiency of Active Learning

Active learning proved exceptionally efficient. The system matched the performance of the full supervised baseline using only 6.6% of clinical labels and 5.8% of newswire labels. Uncertainty sampling naturally favored positive examples in skewed datasets, effectively balancing the training data.

Conclusion

This work demonstrates that while clinical text is inherently complex and "telegraphic," domain-specific features combined with semi-supervised and active learning can significantly lower the barrier to entry for specialized TE tasks. Future work will likely explore deep learning architectures to further improve the absolute performance in these challenging domains.

Summary updated

BibTeX
@inproceedings{shivade2016addressing,
  author = {Chaitanya P. Shivade and Preethi Raghavan and Siddharth Patwardhan},
  title = {{Addressing Limited Data for Textual Entailment Across Domains}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2016},
  doi = {10.18653/v1/p16-1118},
}
Figure