Active Sampling for Text Classification with Subinstance Level Queries

Shayok Chakraborty, Ankita SinghAAAI 2022 · OpenReview

Abstract

Active learning (AL) algorithms effectively identify salient samples from large unlabeled datasets, significantly reducing human annotation effort. In domains like text mining, human oracles often process data incrementally to determine class labels. It is often unnecessary to review a sample end-to-end; identifying an optimal subinstance size (e.g., the first 20% of a review) can be more efficient. This paper proposes a novel framework that jointly selects a batch of informative samples and their optimal subinstance sizes. By posing this as a constrained optimization problem with a linear programming relaxation, the framework maximizes information gain within a fixed labeling budget. Empirical studies on six datasets demonstrate the framework's superiority over existing baselines.

Introduction

Active Learning (AL) aims to minimize the labeling burden by selecting the most informative data points for human annotation. However, in text or video classification, the cost of labeling a single instance can be high if the annotator must consume the entire content. Previous research into "Anytime Active Learning" (AAL) suggests that humans often form accurate opinions after seeing only a fraction of a sample.

While promising, existing AAL methods are typically "myopic," selecting one sample at a time. This is computationally inefficient and incompatible with modern batch-oriented crowdsourcing workflows. This paper addresses these limitations by introducing a batch-mode subinstance selection framework that optimizes both sample choice and snippet length simultaneously.

Methodology

The proposed framework treats sample and snippet-length selection as a joint optimization problem consisting of three main components:

  1. Active Sample Selection: The system evaluates samples based on Uncertainty (using Shannon entropy) and Diversity (using a kernelized distance matrix) to ensure the selected batch is both informative and non-redundant.
  2. Active Subinstance Selection: To prevent querying snippets that are too short to be meaningful, the authors introduce a Neutrality Model. This binary SVM predicts the probability that an annotator will be able to provide a label for a given snippet size, acting as a guardrail against uninformative, overly brief queries.
  3. Optimization Engine: The selection process is formulated as an integer programming problem, which is then simplified using a Linear Programming (LP) relaxation. This allows the system to efficiently identify a batch of (sample, length) pairs that maximize model improvement while adhering to a total cost budget $B$.

Experimental Evaluation

The framework was evaluated on six diverse text classification datasets, including IMDB, SRAA, and Hotel Reviews. The authors utilized TF-IDF features and a Logistic Regression base model to maintain consistency with prior literature.

Test accuracy on the IMDB dataset across iterations.

As shown in the IMDB results, the proposed joint optimization approach (the black line) reaches higher accuracy levels much earlier than baselines like random sampling or standard BatchRank. Similarly, on the SRAA dataset, the framework maintains a significant performance lead throughout the iteration process.

Test accuracy on the SRAA dataset across iterations.

The method proved robust across various budget levels and snippet granularity settings. By optimizing snippet size, the model can "see" more unique samples for the same total cost, leading to better generalization and faster convergence compared to methods that use fixed or myopically chosen subinstance sizes.

Conclusion

This research presents a scalable, batch-mode framework for subinstance-level active learning. By modeling the trade-off between snippet cost and label reliability, the system significantly reduces the human effort required for text classification. Future work will explore integrating this framework with deep learning architectures and addressing the challenges of noisy oracles in real-world labeling environments.

Summary updated

Abstract

Active learning algorithms are effective in identifying the salient and exemplar samples from large amounts of unlabeled data. This tremendously reduces the human annotation effort in inducing a machine learning model as only a few samples, which are identified by the algorithm, need to be labeled manually. In problem domains like text mining and video classification, human oracles peruse the data instances incrementally to derive an opinion about their class labels (such as reading a movie review progressively to assess its sentiment). In such applications, it is not necessary for the human oracles to review an unlabeled sample end-to-end in order to provide a label; it may be more efficient to identify an optimal subinstance size (percentage of the sample from the start) for each unlabeled sample, and request the human annotator to label the sample by analyzing only the subinstance, instead of the whole data sample. In this paper, we propose a novel framework to address this challenging problem, in an effort to further reduce the labeling burden on the human oracles and utilize the available labeling budget more efficiently. We pose the sample and subinstance size selection as a constrained optimization problem and derive a linear programming relaxation to select a batch of exemplar samples, together with the optimal subinstance size of each, which can potentially augment maximal information to the underlying classification model. Our extensive empirical studies on six challenging datasets from the text mining domain corroborate the practical usefulness of our framework over competing baselines.
BibTeX
@inproceedings{chakraborty2022active,
  author = {Shayok Chakraborty and Ankita Singh},
  title = {{Active Sampling for Text Classification with Subinstance Level Queries}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=QxlkIqNv_M},
}
Figure