Introduction
Self-Paced Learning (SPL) is a training paradigm inspired by human education: starting with simple concepts and gradually introducing more complex ones. In machine learning, this translates to training on "easy" samples (those with low loss) first and progressively including "harder" samples as the model matures. This strategy is particularly effective for building robust models that are resilient to noisy or corrupted data.
However, scaling SPL to massive datasets presents a significant challenge. In a standard centralized setup, the weight of every data point is coupled with the global model state. This dependency makes it difficult to distribute training across multiple machines without sacrificing the "global view" of sample difficulty. Naive data splitting often leads to local models misidentifying hard samples as easy, degrading overall performance. This paper introduces Distributed Self-Paced Learning (DSPL), a framework that leverages the Alternating Direction Method of Multipliers (ADMM) to decouple these dependencies, enabling parallel optimization while maintaining global consistency.
Methodology: Distributed Self-Paced Learning
The core innovation of DSPL lies in reformulating the SPL objective into a distributed optimization problem using Consensus ADMM.
Decoupling with Auxiliary Variables
Instead of optimizing a single global parameter vector $w$, DSPL assigns a local parameter $w_i$ to each data batch. To ensure these local models converge to a unified solution, a global consensus variable $z$ is introduced. The optimization problem is constrained such that all local models must eventually align with the global one ($w_i = z$).
Parallel Optimization via ADMM
By introducing the consensus variable, the batches are linked only through $z$. This allows the model parameters ($w_i$) and the instance weights ($v_i$) for each batch to be updated in parallel across different computing nodes. The DSPL algorithm alternates between three main steps:
- Local Model Update: Updating $w_i$ for each batch in parallel.
- Global Consensus Update: Aggregating local results to update the global variable $z$.
- Weight Update: Updating local instance weights $v_i$ based on the current "age" of the model, controlled by the pacing parameter $\lambda$.
The parameter $\lambda$ is crucial as it determines the threshold for "easiness." As training progresses, $\lambda$ is increased, allowing the model to incorporate harder samples. The authors provide a theoretical proof that this distributed approach converges to a critical point, even under non-convex conditions.
Experimental Setup
The researchers evaluated DSPL against several robust regression baselines, including Torrent, DRLR, and standard SPL, using both synthetic and real-world datasets.
- Synthetic Data: Datasets were generated with corruption ratios ranging from 10% to 90%. Adversarial noise was added to labels to test the robustness of the recovery process.
- Real-World Data (Airbnb): The team used house rental price datasets from New York City (~650k samples) and Los Angeles (~210k samples). The task involved predicting rental prices based on 21 features in the presence of artificial price corruption.
Key Results and Discussion
DSPL demonstrates significant advantages in both robustness and efficiency.
Robustness to High Corruption
In synthetic tests, DSPL consistently outperformed other robust regression methods. As shown in Figure 1, while competitors' errors spiked as the corruption ratio increased, DSPL maintained a remarkably low $L_2$ error. Even at 90% corruption, DSPL achieved an error of approximately 0.045.
Real-World Performance
On the NYC Airbnb dataset with 90% corruption, DSPL achieved a Mean Absolute Error (MAE) of 6.444. This is a substantial improvement over standard SPL (7.283) and other distributed baselines, which often exceeded an MAE of 8.3. Furthermore, DSPL exhibited lower variance across multiple runs, suggesting higher reliability for real-world deployment.
Efficiency and Scalability
By allowing local parameters and weights to be calculated independently, DSPL removes the computational bottleneck of traditional SPL. The consensus mechanism ensures that the model maintains a "global view" of sample difficulty, preventing local batches from being misled by outliers.
Conclusion and Future Work
DSPL successfully scales the self-paced learning strategy to large-scale distributed environments. While the current implementation shows promise, some challenges remain. The communication overhead required for ADMM synchronization can be a bottleneck in slow networks. Additionally, the "pacing" parameters ($\lambda$ and its growth rate) currently require manual tuning. Future work could explore automating hyperparameter selection and extending the DSPL framework to massive-scale Deep Neural Networks (DNNs).
Summary updated