Abstract
Supervised learning for text categorization typically requires a large volume of labeled training data, which is expensive to obtain. This paper presents an active learning method that utilizes a committee of learners to selectively sample informative examples. Using a committee of Winnow-based algorithms, the authors demonstrate that this approach can reduce the number of required labeled training examples by 1–2 orders of magnitude compared to standard passive learning, achieving high performance with only 2.9% of the available training labels.
Introduction
In the late 1990s, the rapid growth of electronic text necessitated automated categorization systems. However, text categorization presents a "messy" domain for machine learning, characterized by high-dimensional feature spaces (often exceeding 10,000 unique words), significant noise, and many irrelevant features. The primary bottleneck in deploying these systems is the cost of human-labeled data. This research explores "Active Learning"—where the algorithm itself selects which data points are most beneficial to learn from—to minimize human intervention without compromising accuracy.
Methodology: Active Learning with Committees
The proposed method is inspired by the "Query by Committee" (QBC) framework. It employs a small committee of seven Winnow algorithms. Winnow is a linear threshold algorithm particularly effective for text because its multiplicative weight updates allow it to handle irrelevant attributes efficiently.
Selective Sampling Process
- Committee Initialization: A committee of seven learners is maintained.
- Unlabeled Data Stream: The system is presented with a stream of unlabeled documents.
- Disagreement as a Signal: To determine if a label is needed, two committee members are chosen at random to predict the document's category.
- Querying: If the two members disagree, the document is deemed "informative" (likely residing near the decision boundary), and the system requests a label from a human expert.
- Updating: Upon receiving a label, all committee members update their internal decision boundaries.
The researchers compared four configurations: active-majority (the proposed method), passive-majority (standard committee learning), active-single, and passive-single (the baseline).
Experimental Setup
Experiments were conducted on the Reuters-22173 corpus, a benchmark of 22,173 news articles with 21,334 unique tokens. The study focused on the 10 most frequent topic categories. To simulate real-world conditions, minimal preprocessing was performed on article titles. The training pool consisted of 21,000 examples, with 1,173 reserved for testing.
Key Results
The results demonstrate a dramatic reduction in the need for labeled data. The active-majority system achieved the same final accuracy as standard supervised learning while using only 2.9% of the labeled training examples.
As shown in the learning curves above, active learners reach high accuracy levels much earlier in the training process. While all systems eventually converge to a similar accuracy (approximately 95-96%), the active learner identifies the most "useful" labels early, hitting a performance plateau rapidly.
Furthermore, the active-majority system proved computationally efficient. Despite the overhead of managing a committee, it was faster to train than a passive committee because it performed far fewer expensive weight updates on redundant, non-informative data.
Conclusion and Future Work
This work proves that a small committee (just seven members) is sufficient to achieve massive gains in data efficiency. The choice of the Winnow algorithm was critical due to its robustness against irrelevant features common in text data.
Future research directions include moving from one-by-one queries to batching the top N most informative examples for more practical human interaction, and applying active learning to search engines for more effective relevance feedback.
Summary updated