Learning to Learn Gradient Aggregation by Gradient Descent

Jinlong Ji, Xuhui Chen, Qianlong Wang, Lixing Yu, Pan LiIJCAI 2019 · DOI

TL;DR

This paper introduces a novel "learning to learn" approach for distributed machine learning, where a recurrent neural network (RNN) learns to aggregate gradients from worker nodes. This meta-learning strategy allows for more robust and scalable distributed training, outperforming traditional methods, especially in the presence of Byzantine attacks.

What Problem Does This Paper Address?

The paper tackles two major challenges in distributed machine learning: robustness and scalability. In large-scale distributed systems, where data is spread across many machines (workers) and processed by a central server (master), aggregating local model updates (gradients) from workers is crucial. Traditional methods, like simple averaging, are often deterministic and brittle. They struggle with:

  • Fault tolerance: Workers can fail or be compromised by attackers (Byzantine attacks), sending malicious gradients that can derail the entire learning process.
  • Scalability and heterogeneity: Existing methods typically assume all workers have identical local models. However, in real-world scenarios, workers might have varying computational power or access to different model complexities, limiting the system's ability to scale.

The authors propose a "learning to learn" paradigm, where the aggregation process itself is treated as a learning problem, allowing the system to adapt and improve its aggregation strategy over time.

Methods

The core of the paper is the development of a Recurrent Neural Network (RNN) based aggregator. Instead of using a fixed aggregation rule, the parameter server employs an RNN to learn how to combine gradients from workers. This RNN acts as a meta-learner, observing past aggregation experiences to make better decisions.

Key methodological contributions include:

  • RNN Aggregator: A recurrent neural network (specifically LSTM or GRU variants are mentioned) is used to process the gradients from multiple workers. The RNN learns to store and utilize information from previous aggregation steps to predict the next global model update.
  • Coordinatewise Preprocessing and Postprocessing: To make the RNN aggregator more robust to variations in gradient magnitudes (due to different data shards or worker capabilities), the authors introduce methods to rescale input gradients and output model updates. This helps stabilize the learning process.
  • ARNN Aggregator (Additional Loss Information): To specifically address Byzantine attacks, an enhanced RNN aggregator (ARNN) is proposed. This version incorporates additional loss information calculated from a small subset of public data. This extra signal helps the aggregator distinguish between honest and malicious gradients, particularly when attackers form a majority.
  • Learning to Learn Framework: The entire aggregation process is framed as a meta-learning task. The RNN aggregator is trained by minimizing a loss function that spans multiple optimization steps (iterations) of the distributed learning process, using Back-propagation Through Time (BPTT).

Datasets

The paper evaluates its methods on two standard image classification datasets:

  • MNIST: A dataset of handwritten digits.
  • CIFAR-10: A dataset of 10 classes of images.

For generalization experiments, they also create modified versions of CIFAR-10:

  • CIFAR-2: Contains data for only 2 of the original labels.
  • CIFAR-5: Contains data for 5 of the original labels.

Key Insights and Ideas

  • Treating aggregation as a learning problem: The most significant insight is shifting from hand-designed aggregation rules to a learned, adaptive approach. This allows the system to discover complex aggregation strategies that are difficult to engineer manually.
  • RNNs as powerful aggregators: The use of RNNs, with their inherent ability to process sequential data and maintain memory, proves effective for learning temporal dependencies in the gradient aggregation process.
  • Generalization is achievable: The learned aggregators demonstrate remarkable generalization capabilities, performing well even when transferred to different worker models and datasets than those they were initially trained on. This suggests the learned aggregation strategy captures fundamental principles of distributed learning.
  • Additional loss information for robustness: Incorporating auxiliary loss information provides a crucial signal for identifying and mitigating the impact of malicious gradients, significantly improving resilience against Byzantine attacks.

Key Results

  • Outperforms averaging: Both the RNN and ARNN aggregators consistently outperform the classical averaging aggregator on MNIST and CIFAR-10 datasets.
  • Competitive with Krum (fault tolerance): The ARNN aggregator shows competitive performance against Krum, a state-of-the-art fault tolerance algorithm.
  • Significant improvement against Byzantine attacks: Under majoritarian Byzantine attacks, the ARNN aggregator outperforms Krum by a substantial 43.14%. This is attributed to its ability to learn to identify honest gradients, whereas Krum can be fooled by attackers dominating the majority.
  • Scalability with heterogeneous models: The RNN aggregator successfully utilizes gradients from workers with different local model architectures, demonstrating improved scalability for distributed learning systems with heterogeneous workers.

Limitations and Future Work

  • Computational overhead: While effective, training an RNN aggregator introduces additional computational cost at the parameter server compared to simple averaging.
  • Byzantine attack assumptions: The ARNN aggregator's effectiveness against Byzantine attacks relies on the availability of a public dataset for calculating additional loss information. The paper focuses on majoritarian attacks, and performance against other attack types might differ.
  • Future Work: The authors suggest that the RNN aggregator can work hierarchically and potentially manage multiple distributed systems. They also hint at its potential for improving learning performance by utilizing information from other distributed learning systems. The ARNN aggregator's resilience to Byzantine attackers is highlighted, suggesting further exploration in this area.

Summary updated

BibTeX
@inproceedings{ji2019learning,
  author = {Jinlong Ji and Xuhui Chen and Qianlong Wang and Lixing Yu and Pan Li},
  title = {{Learning to Learn Gradient Aggregation by Gradient Descent}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/363},
}
Figure