Taking into Account the Differences between Actively and Passively Acquired Data: The Case of Active Learning with Support Vector Machines for Imbalanced Datasets

Michael Bloodgood, K. Vijay-ShankerNAACL 2009

TL;DR

This paper tackles the challenge of imbalanced datasets in active learning (AL) for Support Vector Machines (SVMs). They introduce a novel method, InitPA, that sets cost parameters for SVMs based on the overall corpus imbalance, rather than just the imbalanced labeled data, leading to improved performance in both sampling and prediction.

What Problem Does This Paper Address?

Active learning (AL) is a powerful technique for reducing the human effort required to label data for machine learning models. However, most AL research assumes that the data acquired through active sampling behaves similarly to data acquired passively (i.e., randomly). This paper highlights a crucial difference: actively sampled data can have distinct characteristics. This is particularly problematic in domains like Human Language Technology (HLT) where datasets are often imbalanced (one class is much rarer than others) and SVMs are used. Standard passive learning approaches to handle imbalance might not work well when applied to actively sampled data, potentially leading to suboptimal models. The authors specifically focus on imbalanced datasets and SVMs, a common and important scenario in HLT.

Methods

The core of the paper is the InitPA method, designed to address class imbalance in Active Learning with Cost-Weighted SVMs (cwSVMs).

  • Cost-Weighted SVMs (cwSVMs): These are standard SVMs that allow for unequal costs when misclassifying different classes. This is crucial for imbalanced data, as it allows the model to penalize errors on the rare (positive) class more heavily. The cost ratio ($C^+/C^-$) is a key parameter.
  • Passive Learning (PL) Heuristic: In passive learning, a common heuristic for setting the cost ratio is to make it proportional to the ratio of negative to positive examples in the entire training dataset.
  • Active Learning (AL) Challenge: The problem arises because AL systematically selects data points that are close to the current decision boundary. This can lead to the labeled training data becoming skewed, with a higher proportion of positive examples than present in the overall corpus. Applying the PL heuristic directly to this skewed labeled data during AL can lead to underestimating the true imbalance.
  • CurrentPA (Baseline AL Method): This method sets the cost ratio ($PA = C^+/C^-$) based on the imbalance observed in the currently labeled training data at each iteration of the AL loop.
  • InitPA (Proposed Method): This is the paper's main contribution. Instead of re-calculating the cost ratio at each AL iteration based on the skewed labeled data, InitPA estimates the overall corpus imbalance using a small, initial unbiased sample of labeled data. This estimated ratio is then used to set the cost ratio ($PA$) once and applied consistently throughout the entire AL process. This prevents the cost model from being misled by the AL sampling strategy.

Datasets

The authors evaluate their methods on two types of datasets:

  • Relation Extraction (RE): The AImed dataset was used. This dataset was previously used for training protein interaction extraction systems. For this paper, RE was cast as a binary classification task.
  • Text Classification (TC): The Reuters-21578 ModApte split was used. For this dataset, each category was treated as a separate binary classification problem, a common practice for handling multi-class imbalanced data. The ten largest categories were used, exhibiting imbalances ranging from 1.88% to 29.96%.

No new benchmarks were introduced; existing, well-established datasets were utilized.

Key Insights and Ideas

  • Active Sampling Skews Data: The most critical insight is that active learning, by its nature, tends to select data points that are close to the decision boundary. This can lead to a biased sample of labeled data that doesn't accurately reflect the overall class distribution of the corpus.
  • PL Imbalance Strategies Fail in AL: Directly applying passive learning heuristics for handling imbalance (which rely on the overall training data distribution) to the actively sampled, potentially skewed, labeled data during AL can lead to suboptimal performance.
  • Corpus-Level Imbalance is Key: The true imbalance of the entire dataset (corpus) is more important for setting cost parameters in AL-SVMs than the imbalance in the currently labeled subset.
  • InitPA: A Robust Solution: Estimating the corpus imbalance once using an initial unbiased sample and using that estimate consistently throughout AL provides a more stable and effective cost model.
  • InitPA Benefits Both Sampling and Prediction: The paper found that the hyperplanes learned using the InitPA cost model were better for both selecting informative data points (sampling) and for making final predictions.

Key Results

  • InitPA Outperforms: InitPA consistently outperformed other methods, including CurrentPA (which uses the labeled data imbalance) and oversampling techniques, across both AImed and Reuters datasets.
  • Practical Gains: The improvements were particularly noticeable in the "practically important area" where learning curves begin to plateau, suggesting that InitPA leads to better final models.
  • Reuters Nuance: For the Reuters dataset, the gains of InitPA over CurrentPA were smaller. The authors explain this by noting that in some cases, the CurrentPA method might already achieve near-perfect recall on the training data, leaving less room for improvement from a more accurate cost model.
  • Hybrid Approaches Not Better: Interestingly, hybrid approaches where InitPA was used for sampling and CurrentPA for prediction (or vice-versa) did not yield better results than using InitPA for both tasks.

Limitations and Future Work

  • Initial Sample Size: The paper acknowledges that determining the appropriate size for the initial unbiased sample is important. They suggest using standard statistical techniques to ensure a desired level of confidence in the estimated corpus imbalance. For their experiments, an initial labeled set of size 100 was used.
  • Generalizability of InitPA: While InitPA is shown to be effective for AL-SVMs with imbalanced data, the authors suggest that the broader idea of modifying inference algorithms to suit the unique characteristics of actively sampled data is an area ripe for further investigation. This could extend beyond SVMs and imbalanced datasets.
  • Stopping Criteria: The paper mentions that the AL process stops when predictions stabilize (Bloodgood and Vijay-Shanker, 2009), but the focus here is on the cost model during AL, not the stopping criteria itself.

Summary updated

BibTeX
@inproceedings{bloodgood2009taking,
  author = {Michael Bloodgood and K. Vijay-Shanker},
  title = {{Taking into Account the Differences between Actively and Passively Acquired Data: The Case of Active Learning with Support Vector Machines for Imbalanced Datasets}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2009},
}
Figure