Distributed Self-Paced Learning in Alternating Direction Method of Multipliers

Xuchao Zhang, Liang Zhao, Zhiqian Chen, Chang-Tien Lu

Introduction

Self-Paced Learning (SPL) is a robust machine learning paradigm inspired by the human educational process of curriculum learning. In SPL, a model begins training on "easy" samples—those with low loss—and gradually incorporates "harder," higher-loss samples as its maturity increases. This approach is particularly effective at mitigating the influence of outliers and corrupted data, which typically exhibit high loss early in training.

However, traditional SPL faces significant scalability challenges. Determining which samples are "easy" requires a global view of the entire dataset, creating a computational bottleneck. Massive datasets often exceed the memory capacity of a single machine, and the global dependency between instance weights and the model parameters prevents simple batch-wise parallelization. Without global context, a batch containing only corrupted data might incorrectly identify those outliers as "easy" samples.

The DSPL Framework

To address these limitations, this paper introduces Distributed Self-Paced Learning (DSPL). The core innovation lies in reformulating the SPL objective into a distributed optimization problem using the Consensus Alternating Direction Method of Multipliers (ADMM).

Decoupling via Consensus

Instead of maintaining a single global model, DSPL assigns a local model ($w_i$) to each data batch. To prevent these local models from diverging, the framework introduces a global auxiliary variable ($z$) and enforces a constraint that all local models must eventually converge to this global consensus ($w_i = z$). This decoupling allows for a three-step parallel optimization loop:

  1. Parallel Model Update: Each node updates its local model ($w_i$) based on its local data and the current global consensus.
  2. Global Consensus: A central variable ($z$) is updated by averaging the local models, serving as a synchronization bridge.
  3. Parallel Weight Update: Each node independently updates its local instance weights ($v_i$), identifying "easy" samples relative to the current local model.

This structure allows the "age" of the learner—controlled by the parameter $\lambda$—to advance across the entire distributed system simultaneously.

Experimental Evaluation

The researchers evaluated DSPL against several baselines, including Torrent (TORR), RLHH, and DRLR, using both synthetic and real-world datasets. Synthetic tests involved varying dimensions ($p$) and sample sizes ($n$), with adversarial noise injected into up to 90% of the batches.

Recovery error across different corruption ratios and dataset configurations.

Key Results

  • Robustness to Heavy Corruption: In synthetic tests with 90% corrupted batches, DSPL maintained a low $L_2$ error (approximately 0.045), significantly outperforming standard robust algorithms like TORR. As shown in the figures, while competitors' error rates spiked as corruption passed 40%, DSPL's performance remained remarkably stable.
  • Real-World Performance: On large-scale Airbnb regression tasks (predicting rental prices in NYC and LA), DSPL consistently achieved lower Mean Absolute Error (MAE) than traditional SPL and other robust regression baselines, demonstrating its ability to handle the inherent noise of real-world market data.

Discussion and Limitations

While DSPL offers a scalable and robust solution for distributed learning, it is not without challenges. The method is sensitive to the "age" parameter $\lambda$ and its growth rate $\mu$. Finding the optimal values for these hyperparameters remains a manual process, though the relationship between the Lagrangian and recovery error provides a clear "sweet spot" for optimization.

The relationship between recovery error, the Lagrangian, and the learning pace parameter λ.

Furthermore, while the ADMM framework is computationally efficient due to parallelization, it introduces communication overhead because the global variable $z$ must be synchronized across nodes. Future work may explore the application of DSPL to more complex architectures, such as Transformers or CNNs, beyond the robust regression tasks focused on in this study.

Summary updated

Abstract

Self-paced learning (SPL) mimics the cognitive process of humans, who generally learn from easy samples to hard ones. One key issue in SPL is the training process required for each instance weight depends on the other samples and thus cannot easily be run in a distributed manner in a large-scale dataset. In this paper, we reformulate the self-paced learning problem into a distributed setting and propose a novel Distributed Self-Paced Learning method (DSPL) to handle large scale datasets. Specifically, both the model and instance weights can be optimized in parallel for each batch based on a consensus alternating direction method of multipliers. We also prove the convergence of our algorithm under mild conditions. Extensive experiments on both synthetic and real datasets demonstrate that our approach is superior to those of existing methods.
BibTeX
@inproceedings{zhang2018distributed,
  author = {Xuchao Zhang and Liang Zhao and Zhiqian Chen and Chang-Tien Lu},
  title = {{Distributed Self-Paced Learning in Alternating Direction Method of Multipliers}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.24963/ijcai.2018/437},
}
Figure