Self-Paced Boost Learning (SPBL) is a unified learning framework introduced at IJCAI-16 that integrates the discriminative power of ensemble boosting with the robustness of self-paced learning (SPL). By iteratively selecting "easy" (reliable) samples before tackling "hard" (potentially noisy) ones, SPBL creates a model that is significantly more resistant to label noise and outliers than traditional ensemble methods.
The Challenge of Noise in Supervised Learning
In supervised learning, there is a constant tension between effectiveness and robustness. Boosting algorithms like AdaBoost are excellent at fitting complex, non-linear data by focusing on samples that the current model misclassifies. However, this makes them "noise-sensitive," as they often overfit to outliers or mislabeled data. Conversely, Self-Paced Learning mimics human education by starting with simple concepts and gradually increasing difficulty. While robust, SPL alone may not capture the complex inter-class boundaries required for high-accuracy classification.
Methodology
The core innovation of SPBL is its objective function, which treats sample selection and model boosting as a joint optimization problem. The algorithm uses a max-margin boosting formulation but weights each training sample with an "easiness" variable ($v_i$).
Unlike standard boosting, where weights are determined solely by classification error, SPBL weights are a composite of learning insufficiency (how much the model struggles with the point) and learning reliability (how likely the point is to be a valid sample). The optimization follows a two-step iterative process:
- Update Weights ($v$): Fix the model and select the "easiest" samples based on the current loss and a "pace" parameter ($\lambda$).
- Update Model ($W$): Fix the samples and use Column Generation to add the best weak learner to the ensemble.
As iterations progress, the pace parameter is annealed, allowing the model to incorporate increasingly complex samples once it has mastered the simpler patterns.
Experimental Evaluation and Results
The authors evaluated SPBL on three image datasets: Caltech256, AnimalWithAttributes (AWA), and Corel10k. To test robustness, they manually injected label noise by randomly relabeling 5%, 10%, and 15% of the training data.
Key Findings
- Superiority in Noise: In every test case, SPBL outperformed standard Boosting (MultiBoost, AdaBoost) and standard robust models. For example, on the Caltech dataset with 15% noise, SPBL achieved an error rate of 0.4058, significantly better than MultiBoost (0.4361) and AdaBoost (0.4298).
- Generalization Stability: A major finding was that SPBL maintains a much smaller gap between training error and test error compared to MultiBoost. This suggests that the "smooth" learning pace acts as a powerful regularizer, preventing the model from chasing noise.
- Reliability Filtering: In early stages, SPBL effectively ignores outliers because their loss is too high to be considered "easy." By the time the pace increases to include them, the model is already strong enough to recognize them as anomalies rather than core patterns.
Conclusion
SPBL successfully bridges the gap between discriminative boosting and robust self-paced learning. While the alternating optimization approach introduces some computational overhead and requires hyperparameter tuning, its ability to maintain performance under high label noise makes it a robust alternative for real-world classification tasks where data quality cannot be guaranteed.
Summary updated