Abstract
Textual Entailment (TE) is a fundamental NLP task that determines if a hypothesis logically follows from a premise. While successful in general domains, TE faces significant hurdles in specialized fields like medicine due to domain shifts and the high cost of expert annotation. This paper introduces a clinical dataset and evaluates strategies—specifically self-training and active learning—to mitigate data scarcity. Our findings show that these methods can improve F-scores by up to 15% using unlabeled data or achieve baseline performance using less than 7% of the total labeled dataset.
Introduction: The Challenge of Specialized Domains
Textual Entailment (TE) systems are typically trained on newswire data, which does not translate well to the medical domain. Specialized fields present two primary challenges: a significant domain gap in terminology and structure, and a severe scarcity of labeled data. Annotating medical records requires clinical expertise and is often restricted by privacy regulations.
The authors focus on entailment search, which involves identifying all sentences in a corpus that support a specific clinical hypothesis, such as "Does this patient have a history of hypertension?"
Methodology
The research employs a three-pronged approach to bridge the data gap:
- Supervised Baseline (ENT): A logistic regression classifier utilizing lexical, syntactic, and domain-specific features. Key to its success is the integration of the Unified Medical Language System (UMLS) to handle medical concepts and a specialized translator for mapping jargon to common terms.
- Self-Training: This semi-supervised method leverages unlabeled data. The model iteratively labels unlabeled pairs; those exceeding a confidence threshold ($\tau$) are added to the training set for subsequent rounds.
- Active Learning: Using uncertainty sampling, the system identifies instances where the classifier is most uncertain (probabilities near 0.5). Human experts then label only these high-value instances, maximizing the information gain per label.
Data and Experimental Setup
The study compares two domains:
- Newswire: Standard datasets from PASCAL RTE challenges.
- Clinical: A novel dataset derived from Electronic Medical Records (EMRs), where medical students annotated the relationship between clinical hypotheses and patient notes.
- Unlabeled Resources: Large-scale unlabeled corpora were used, including ~43k newswire pairs and ~623k clinical pairs.
Results and Discussion
The experimental results highlight the effectiveness of leveraging unlabeled data and strategic sampling.
The Impact of Self-Training
Self-training provided substantial performance boosts. The F-score increased by 15% on newswire (0.39 to 0.54) and 13% on clinical data (0.23 to 0.36). A critical finding was the sensitivity of the system to the confidence threshold ($\tau$).
As shown in the figures above, lower thresholds tend to increase recall but introduce noise, while higher thresholds maintain precision at the cost of diversity.
Efficiency of Active Learning
Active learning proved exceptionally efficient. The system matched the performance of the full supervised baseline using only 6.6% of clinical labels and 5.8% of newswire labels. Uncertainty sampling naturally favored positive examples in skewed datasets, effectively balancing the training data.
Conclusion
This work demonstrates that while clinical text is inherently complex and "telegraphic," domain-specific features combined with semi-supervised and active learning can significantly lower the barrier to entry for specialized TE tasks. Future work will likely explore deep learning architectures to further improve the absolute performance in these challenging domains.
Summary updated