Active Learning with Committees

Ray Liere, Prasad TadepalliAAAI 1997 · OpenReview

Abstract

In many real-world domains like text categorization, supervised learning requires a large number of training examples. In our research, we are using active learning with committees methods to reduce the number of training examples required for learning. Disagreement among the committee members on the predicted label for the input part of each example is used to determine the need for knowing the actual value of the label. Our experiments in text categorization using this approach demonstrate a 1-2 orders of magnitude reduction in the number of labeled training examples required.

Introduction

Text categorization—the process of assigning documents to predefined categories based on content—poses a significant challenge for traditional machine learning. These tasks typically involve massive feature spaces, often exceeding 10,000 dimensions, characterized by high levels of noise and a vast majority of irrelevant features.

The primary bottleneck in developing these systems is not computational power, but the "labeling cost." Standard supervised learning requires human experts to manually label thousands of examples to train a reliable model. This paper addresses the problem of "label complexity": how to train a high-performing classifier using the absolute minimum number of human-provided labels.

Methodology: Query by Committee

The authors build upon the concept of Query by Committee (QBC). In a theoretical QBC framework, one starts with a committee representing all possible hypotheses. A human is only asked for a label when the committee members strongly disagree on a specific example. If the committee agrees, the example is likely redundant, and the system moves on without requiring human intervention.

The Practical Implementation

Instead of a theoretical, infinite committee, this research implements a practical version with the following components:

  1. Small, Finite Committee: The system uses a small group of distinct hypotheses that are updated as learning progresses.
  2. The Winnow Algorithm: Each committee member utilizes the Winnow learning algorithm. Winnow was specifically selected because it excels in high-dimensional spaces where many features are irrelevant—a characteristic inherent to text data.
  3. Active Sampling: When a feature vector is presented, the committee checks for disagreement. High disagreement triggers a request for the actual label. This label is then used to update the committee members, discarding or refining hypotheses that were incorrect.
  4. Majority Voting: For final system predictions, the committee employs a simple majority vote.

Experimental Setup

The researchers evaluated the approach using the Reuters-22173 corpus, a classic benchmark in news categorization. The experiments specifically utilized the titles of the 22,173 documents. This dataset represents the high-dimensional, sparse feature characteristics typical of real-world text classification tasks.

Key Insights

The Informativeness of Disagreement

The core insight of the paper is that disagreement serves as a proxy for information gain. If diverse hypotheses disagree on a label, that specific example likely lies near the "version space" boundary. Resolving that disagreement provides the most significant improvement in model accuracy for the least amount of labeling effort.

Convergence and Efficiency

Early in the process, committee members are diverse and disagree frequently, leading to a high rate of label requests. As the members learn and converge toward the optimal target hypothesis, the distance between label requests increases. This naturally tapers off the human workload as the model matures.

Winnow as a Robust Learner

By pairing active learning with Winnow, the authors leverage an algorithm that is mathematically robust against the "noise" of irrelevant words. This synergy allows the system to handle the primary hurdle of text-based machine learning more effectively than standard algorithms.

Results

The quantitative findings of the study are significant:

  • 97.1% Reduction in Data: The active learning committee achieved the same accuracy as a single supervised learner while using only 2.9% of the labeled examples.
  • Efficiency Gains: The authors report a 1–2 orders of magnitude reduction in the number of labeled training examples required.
  • Superior Online Performance: The committee method provided better accuracy during the learning process compared to other systems. This makes it ideal for "anytime" learning scenarios where training might be interrupted by time or budget constraints.

Conclusion and Future Work

This research demonstrates that Active Learning with Committees can slash the amount of labeled data needed for text categorization by up to 97%. While highly effective, several areas remain for future exploration, including how performance scales with larger committee sizes and whether these gains hold when processing full-text documents rather than just titles. Nonetheless, the results suggest a powerful path forward for reducing the human cost of machine learning in complex domains.

Reference: Liere, R., & Tadepalli, P. (1997). Active Learning with Committees. AAAI-97 Proceedings.

Summary updated

Abstract

Active Learning with Committees Ray Liere, Prasad Tadepalli In many real-world domains like text categorization, supervised learning requires a large number of training examples. In our research, we are using active learning with committees methods to reduce the number of training examples required for learning. Disagreement among the committee members on the predicted label for the input part of each example is used to determine the need for knowing the actual value of the label. Our experiments in text categorization using this approach demonstrate a 1-2 orders of magnitude reduction in the number of labeled training examples required. This page is copyrighted by AAAI. All rights reserved. Your use of this site constitutes acceptance of all of AAAI's terms and conditions and privacy policy.
BibTeX
@inproceedings{liere1997active,
  author = {Ray Liere and Prasad Tadepalli},
  title = {{Active Learning with Committees}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1997},
  url = {https://openreview.net/forum?id=SyVCHlZuWH},
}
Figure