Active Learning with Committees for Text Categorization

Ray Liere, Prasad TadepalliAAAI 1997 · OpenReview

Abstract

Supervised learning for text categorization typically requires a large volume of labeled training data, which is expensive to obtain. This paper presents an active learning method that utilizes a committee of learners to selectively sample informative examples. Using a committee of Winnow-based algorithms, the authors demonstrate that this approach can reduce the number of required labeled training examples by 1–2 orders of magnitude compared to standard passive learning, achieving high performance with only 2.9% of the available training labels.

Introduction

In the late 1990s, the rapid growth of electronic text necessitated automated categorization systems. However, text categorization presents a "messy" domain for machine learning, characterized by high-dimensional feature spaces (often exceeding 10,000 unique words), significant noise, and many irrelevant features. The primary bottleneck in deploying these systems is the cost of human-labeled data. This research explores "Active Learning"—where the algorithm itself selects which data points are most beneficial to learn from—to minimize human intervention without compromising accuracy.

Methodology: Active Learning with Committees

The proposed method is inspired by the "Query by Committee" (QBC) framework. It employs a small committee of seven Winnow algorithms. Winnow is a linear threshold algorithm particularly effective for text because its multiplicative weight updates allow it to handle irrelevant attributes efficiently.

Selective Sampling Process

  1. Committee Initialization: A committee of seven learners is maintained.
  2. Unlabeled Data Stream: The system is presented with a stream of unlabeled documents.
  3. Disagreement as a Signal: To determine if a label is needed, two committee members are chosen at random to predict the document's category.
  4. Querying: If the two members disagree, the document is deemed "informative" (likely residing near the decision boundary), and the system requests a label from a human expert.
  5. Updating: Upon receiving a label, all committee members update their internal decision boundaries.

The researchers compared four configurations: active-majority (the proposed method), passive-majority (standard committee learning), active-single, and passive-single (the baseline).

Experimental Setup

Experiments were conducted on the Reuters-22173 corpus, a benchmark of 22,173 news articles with 21,334 unique tokens. The study focused on the 10 most frequent topic categories. To simulate real-world conditions, minimal preprocessing was performed on article titles. The training pool consisted of 21,000 examples, with 1,173 reserved for testing.

Key Results

The results demonstrate a dramatic reduction in the need for labeled data. The active-majority system achieved the same final accuracy as standard supervised learning while using only 2.9% of the labeled training examples.

Learning curves for all systems. Active learning methods (solid and dotted lines) reach high accuracy significantly faster than passive methods.

As shown in the learning curves above, active learners reach high accuracy levels much earlier in the training process. While all systems eventually converge to a similar accuracy (approximately 95-96%), the active learner identifies the most "useful" labels early, hitting a performance plateau rapidly.

Comparison of elapsed processor time and labeled examples used. Active systems (1 and 3) require orders of magnitude fewer examples than passive systems (2 and 4).

Furthermore, the active-majority system proved computationally efficient. Despite the overhead of managing a committee, it was faster to train than a passive committee because it performed far fewer expensive weight updates on redundant, non-informative data.

Conclusion and Future Work

This work proves that a small committee (just seven members) is sufficient to achieve massive gains in data efficiency. The choice of the Winnow algorithm was critical due to its robustness against irrelevant features common in text data.

Future research directions include moving from one-by-one queries to batching the top N most informative examples for more practical human interaction, and applying active learning to search engines for more effective relevance feedback.

Summary updated

Abstract

In many real-world domains, supervised learning requires a large number of training examples. In this paper, we describe an active learning method that uses a committee of learners to reduce the number of training examples required for learning. Our approach is similar to the Query by Committee framework, where disagreement among the committee members on the predicted label for the input part of the example is used to signal the need for knowing the actual value of the label. Our experiments are conducted in the text categorization domain, which is characterized by a large number of features, many of which are irrelevant. We report here on experiments using a committee of Winnowbased learners and demonstrate that this approach can reduce the number of labeled training examples required over that used by a single Winnow learner by l-2 orders of magnitude. 1. Hntroduction
BibTeX
@inproceedings{liere1997active,
  author = {Ray Liere and Prasad Tadepalli},
  title = {{Active Learning with Committees for Text Categorization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1997},
  url = {https://openreview.net/forum?id=Bk4ctax_-H},
}
Figure