Active Learning with Rationales for Text Classification

Manali Sharma, Di Zhuang, Mustafa BilgicNAACL 2015

TL;DR

This paper introduces a novel framework for text classification that leverages "rationales" – the specific phrases or words that justify a label – to improve classifier performance. By modifying document representations to emphasize these rationales, the authors demonstrate significant gains over traditional methods, even with off-the-shelf classifiers. They also propose a new active learning strategy that specifically exploits these rationales to select the most informative documents for labeling, further boosting efficiency.

What Problem Does This Paper Address?

Manually labeling large datasets for text classification is a time-consuming and expensive process. While semi-supervised and active learning techniques aim to reduce this burden, recent work has explored eliciting richer feedback from annotators, such as "rationales" (the specific text snippets that justify a label) in addition to the label itself. However, a major bottleneck has been the difficulty of integrating this rich, domain-specific knowledge into standard machine learning classifiers. Many existing methods are classifier-specific, limiting their applicability. This paper addresses the need for a general, classifier-agnostic approach to effectively incorporate rationales into text classification and to leverage them for more efficient active learning.

Methods

The core of the paper's contribution lies in two main areas:

  • Incorporating Rationales into Classifier Training (Learning with Rationales - LwR): The authors propose a simple yet effective method to modify the feature representation of a document. When a rationale is provided for a document's label, the features corresponding to the rationale are multiplied by a weight r, while other features are multiplied by a weight o. This effectively emphasizes the rationale's contribution to the document's representation. This approach is designed to be classifier-agnostic, working with any standard classifier that uses feature vectors. The paper provides a theoretical justification for this approach by showing how it leads to a more granular form of regularization in models like logistic regression.
  • Active Learning with Rationales (UNC-PC): Building on the LwR framework, the authors develop a new active learning strategy. Traditional uncertainty sampling selects documents where the classifier is most uncertain. Their proposed "uncertain-prefer-conflict" (UNC-PC) strategy first identifies uncertain documents and then prioritizes those that exhibit "conflict" based on the provided rationales. Specifically, it categorizes unlabeled documents into three types: those with no overlap with existing rationales (Type1), those with overlap with rationales for only one class (Type2), and those with overlap with rationales for both positive and negative classes (Type3). UNC-PC favors Type3 documents, hypothesizing that these "conflicting" documents are more informative for resolving ambiguities.

Datasets

The authors evaluated their methods on four standard text classification datasets:

  • IMDB: 25K movie reviews.
  • SRAA: 48K documents discussing auto or aviation.
  • Nova: A dataset from an active learning challenge, containing 12K documents.
  • WvsH: A subset of the 20 Newsgroups dataset, focusing on "Windows vs. hardware" categories, with 1176 documents.

They experimented with both binary and TF-IDF feature representations for these datasets.

Key Insights and Ideas

  • Rationales as a Powerful Signal: The paper strongly suggests that rationales are not just supplementary information but can significantly enhance classification performance, even when provided by a conservative, simulated annotator.
  • Classifier Agnosticism is Key: The proposed LwR method's ability to work with any off-the-shelf classifier is a major advantage, making it broadly applicable without requiring specialized model architectures.
  • Granular Regularization: The LwR approach can be viewed as a form of instance-specific, feature-level regularization, allowing for more nuanced control over feature importance.
  • Conflict Resolution in Active Learning: The UNC-PC strategy highlights the potential of using domain knowledge (rationales) to guide active learning, moving beyond simple uncertainty to actively seek out documents that present conflicting evidence.
  • Efficiency Gains: The results demonstrate that incorporating rationales can drastically reduce the number of documents needed to achieve a target performance level, leading to substantial time savings for annotators.

Key Results

  • Significant Performance Improvement: LwR consistently outperformed learning without rationales (Lw/oR) across all datasets, classifiers (multinomial Naive Bayes, logistic regression, SVM), and representations (binary and TF-IDF). Figure 1 visually demonstrates these improvements, showing higher AUC scores for LwR.
  • Drastic Reduction in Labeling Effort: For example, achieving a target AUC of 0.95 on the SRAA dataset required 656 documents with Lw/oR, but only 29 documents with LwR – a 22.6x reduction. Similarly, reaching a target AUC of 0.8 on WvsH required 113 documents with Lw/oR versus 13 with LwR.
  • Active Learning Benefits: The UNC-PC active learning strategy, when combined with LwR, showed improvements over traditional uncertainty sampling (UNC) on several datasets and classifiers. Table 2 summarizes paired t-test results, indicating that UNC-PC often leads to significantly better learning curves.

Limitations and Future Work

  • Simulated Annotator: The experiments relied on a simulated human expert. A user study is needed to validate these findings with real annotators and to measure actual time savings.
  • Richer Feedback: The current framework uses single phrases as rationales. Future work could explore allowing annotators to provide richer, more nuanced feedback (e.g., linking specific phrases to specific aspects of the label) to resolve more complex conflicts.
  • Cross-Domain Applicability: While the framework is designed to be general, its application to non-text domains (e.g., medical diagnosis) is an interesting avenue for future research.
  • Decoupling Rationales and Documents: The paper notes that while their approach couples rationales to documents, other feature annotation work decouples them. The performance implications of this decoupling need further empirical testing.

Summary updated

BibTeX
@inproceedings{sharma2015active,
  author = {Manali Sharma and Di Zhuang and Mustafa Bilgic},
  title = {{Active Learning with Rationales for Text Classification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
}
Figure