Unbiased Active Semi-supervised Binary Classification Models

JooChul Lee, Weidong Ma, Ziyang WangIJCAI 2024

Introduction

In modern data science, labeling large datasets is often a significant bottleneck due to the high cost and time required for expert annotation. Active Learning (AL) addresses this by strategically selecting the most informative data points for labeling. However, this non-random selection introduces a "sampling bias," causing standard statistical models trained on these subsets to become biased. Conversely, Semi-supervised Learning (SSL) leverages unlabeled data to enhance model performance but typically assumes that labeled data is sampled randomly—an assumption violated in active learning contexts.

This paper introduces a novel framework, the Actively Improved Augmented Estimating Equation (AI-AEE), which bridges the gap between AL and SSL. The proposed method enables the training of binary classifiers that are both statistically unbiased and significantly more efficient than traditional supervised active learning methods by utilizing the entire data pool.

Methodology

The AI-AEE framework relies on two primary components to achieve its objectives:

  1. Corrective Weighting: To mitigate the bias introduced by non-random active selection, the framework employs a weighting scheme that adjusts for selection probabilities. This ensures the estimator targets the true population parameters.
  2. Imputation Models: The method utilizes an imputation model, built from existing labeled data, to "automatically annotate" the remaining unlabeled pool. This allows the classifier to learn from the global structure of the dataset.

A key theoretical contribution is the proof of double robustness: the final classifier remains unbiased even if the imputation model is misspecified. Furthermore, the authors derive an optimal sampling strategy termed Self-Learning Based Subsampling (SBS). Unlike standard heuristics that focus solely on uncertainty, SBS prioritizes points where the discrepancy between the current classifier and the imputation model is greatest, effectively targeting high-variance regions of the feature space.

Experimental Evaluation

The researchers validated the AI-AEE framework using synthetic simulations and four diverse real-world datasets from the UCI Machine Learning Repository: Bank Marketing, SUSY (high-energy physics), Credit Card Clients, and Purchasing Intention. These datasets represent a range of sample sizes and feature complexities, providing a rigorous testbed for the proposed methodology.

Key Results and Discussion

The experimental results consistently demonstrated the superiority of the AI-AEE approach. As shown in the figure below, the proposed AI-SBS method (combining AI-AEE with SBS sampling) significantly reduces both squared bias and root mean squared error (RMSE) compared to the standard corrective weighting baseline (CW-SBS) across various simulation scenarios.

Comparison of Squared Bias and Root Mean Squared Error (RMSE) across three simulation cases for CW-SBS and AI-SBS methods.

Key insights from the evaluation include:

  • Superior Efficiency: AI-AEE achieves higher accuracy with fewer labeled samples than standard methods. On the large-scale SUSY dataset (200,000 samples), the reduction in MSE was particularly dramatic.
  • Bias Convergence: As the number of data batches increases, the statistical bias of the AI-AEE estimator rapidly approaches zero, validating the theoretical claims of unbiasedness.
  • Synergistic Effects: Active learning improves the imputation model by selecting diverse points, which in turn enhances the final semi-supervised classifier. This feedback loop drives the efficiency gains.

Conclusion and Future Work

The AI-AEE framework provides a robust solution for training unbiased classifiers in active learning settings. While the method is theoretically unbiased regardless of imputation quality, its practical efficiency gains are tied to the accuracy of the imputation model.

Future research directions include addressing the challenges of "rare events" where class imbalance is extreme, potentially through the use of surrogate variables. Additionally, further refinements may be needed to optimize the matrix calculations for extremely high-dimensional feature spaces where the number of predictors exceeds the sample size.

Summary updated

BibTeX
@inproceedings{lee2024unbiased,
  author = {JooChul Lee and Weidong Ma and Ziyang Wang},
  title = {{Unbiased Active Semi-supervised Binary Classification Models}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure