Abstract
Active learning (AL) algorithms effectively identify salient samples from large unlabeled datasets, significantly reducing human annotation effort. In domains like text mining, human oracles often process data incrementally to determine class labels. It is often unnecessary to review a sample end-to-end; identifying an optimal subinstance size (e.g., the first 20% of a review) can be more efficient. This paper proposes a novel framework that jointly selects a batch of informative samples and their optimal subinstance sizes. By posing this as a constrained optimization problem with a linear programming relaxation, the framework maximizes information gain within a fixed labeling budget. Empirical studies on six datasets demonstrate the framework's superiority over existing baselines.
Introduction
Active Learning (AL) aims to minimize the labeling burden by selecting the most informative data points for human annotation. However, in text or video classification, the cost of labeling a single instance can be high if the annotator must consume the entire content. Previous research into "Anytime Active Learning" (AAL) suggests that humans often form accurate opinions after seeing only a fraction of a sample.
While promising, existing AAL methods are typically "myopic," selecting one sample at a time. This is computationally inefficient and incompatible with modern batch-oriented crowdsourcing workflows. This paper addresses these limitations by introducing a batch-mode subinstance selection framework that optimizes both sample choice and snippet length simultaneously.
Methodology
The proposed framework treats sample and snippet-length selection as a joint optimization problem consisting of three main components:
- Active Sample Selection: The system evaluates samples based on Uncertainty (using Shannon entropy) and Diversity (using a kernelized distance matrix) to ensure the selected batch is both informative and non-redundant.
- Active Subinstance Selection: To prevent querying snippets that are too short to be meaningful, the authors introduce a Neutrality Model. This binary SVM predicts the probability that an annotator will be able to provide a label for a given snippet size, acting as a guardrail against uninformative, overly brief queries.
- Optimization Engine: The selection process is formulated as an integer programming problem, which is then simplified using a Linear Programming (LP) relaxation. This allows the system to efficiently identify a batch of (sample, length) pairs that maximize model improvement while adhering to a total cost budget $B$.
Experimental Evaluation
The framework was evaluated on six diverse text classification datasets, including IMDB, SRAA, and Hotel Reviews. The authors utilized TF-IDF features and a Logistic Regression base model to maintain consistency with prior literature.
As shown in the IMDB results, the proposed joint optimization approach (the black line) reaches higher accuracy levels much earlier than baselines like random sampling or standard BatchRank. Similarly, on the SRAA dataset, the framework maintains a significant performance lead throughout the iteration process.
The method proved robust across various budget levels and snippet granularity settings. By optimizing snippet size, the model can "see" more unique samples for the same total cost, leading to better generalization and faster convergence compared to methods that use fixed or myopically chosen subinstance sizes.
Conclusion
This research presents a scalable, batch-mode framework for subinstance-level active learning. By modeling the trade-off between snippet cost and label reliability, the system significantly reduces the human effort required for text classification. Future work will explore integrating this framework with deep learning architectures and addressing the challenges of noisy oracles in real-world labeling environments.
Summary updated