Abstract: We offer a novel view of AdaBoost in a statistical setting. We propose a Bayesian model for binary classification in which label noise is modeled hierarchically. Using variational inference to optimize a dynamic evidence lower bound, we derive a new boosting-like algorithm called VIBoost. We show its close connections to AdaBoost and give experimental results from four datasets.
Introduction
Boosting algorithms, particularly AdaBoost, are fundamental tools in machine learning, known for their ability to aggregate weak classifiers into a strong ensemble. However, AdaBoost is notoriously sensitive to noisy data. Because it iteratively increases the weights of misclassified examples, it often overfits to outliers or incorrectly labeled points. This paper introduces VIBoost (Variational Inference Boosting), a Bayesian reimagining of AdaBoost that explicitly models label noise using hierarchical latent variables, allowing the algorithm to detect and quantify noise rather than being misled by it.
Methods: The VIBoost Framework
VIBoost embeds binary classification into a generative model where labels are treated as either "true" or "noisy." The core of the model involves several key components:
- Versatile Logistic Distribution: A novel conjugate prior for logistic regression that enables efficient posterior calculations.
- Hierarchical Noise Modeling: Latent variables are introduced to represent the data generation process. Specifically, $w_n$ indicates if a label is true or noisy, $\xi$ represents the "noise grade" (log-odds of noise), and $\theta$ is the prior probability of a label being true.
- Stagewise Variational Inference: To maintain the efficiency of boosting, the authors use a stagewise approach, greedily adding base classifiers and updating parameters via variational inference to optimize a dynamic evidence lower bound (ELBO).
This framework provides a principled derivation for techniques like weight shrinkage, which are often applied heuristically in standard AdaBoost to prevent weight explosion. The Bayesian prior naturally introduces a smoothing term that emerges from first principles, rather than ad-hoc adjustments.
Experimental Evaluation and Results
The authors evaluated VIBoost across several datasets, including the UCI Spam dataset and synthetic benchmarks like the Long-Servedio dataset, which is designed to cause convex potential boosters like AdaBoost to fail.
Key findings include:
- Performance Parity: On real-world datasets, VIBoost matches AdaBoost's accuracy, demonstrating that the Bayesian framework does not sacrifice performance on clean data.
- Noise Recovery: In synthetic tests, VIBoost successfully identifies the Signal-to-Noise Ratio (SNR) and recovers the underlying noise grade.
- Robustness to Pathological Data: On the Long-Servedio dataset, while VIBoost struggles to find a perfect classifier, it correctly identifies the high noise level, providing a diagnostic signal that AdaBoost lacks.
Discussion and Conclusion
VIBoost's primary advantage is its ability to treat noise as a diagnostic tool. By providing an SNR estimate, it can distinguish between a poor model fit and a fundamentally noisy dataset. While currently limited to binary classification and assuming conditional independence between instances, VIBoost offers a robust, noise-aware alternative to traditional boosting methods. Future work may extend this approach to multi-class problems and more complex spatial noise dependencies.
Summary updated