Learning to accelerate distributed ADMM using graph neural networks

Henri Doerks, Paul Häusner, Daniel Hernández Escobar, Jens Sjölund

Abstract

Distributed optimization is fundamental in large-scale machine learning and control applications. Among existing methods, the Alternating Direction Method of Multipliers (ADMM) has gained popularity due to its strong convergence guarantees and suitability for decentralized computation. However, ADMM often suffers from slow convergence and sensitivity to hyperparameter choices. In this work, we show that distributed ADMM iterations can be naturally represented within the message-passing framework of graph neural networks (GNNs). Building on this connection, we propose to learn adaptive step sizes and communication weights by a graph neural network that predicts the hyperparameters based on the iterates. By unrolling ADMM for a fixed number of iterations, we train the network parameters end-to-end to minimize the final iterates error for a given problem class, while preserving the algorithm's convergence properties. Numerical experiments demonstrate that our learned variant consistently improves convergence speed and solution quality compared to standard ADMM.

Introduction

Distributed optimization is the backbone of modern large-scale machine learning and control systems. In these environments, data is distributed across a network of agents—such as sensors or mobile devices—that must collaborate to solve a global objective without sharing private local data. The Alternating Direction Method of Multipliers (ADMM) is a widely adopted algorithm for these tasks due to its scalability and flexibility.

Despite its popularity, ADMM faces two primary challenges: slow convergence rates and high sensitivity to hyperparameters. Specifically, the performance of ADMM depends heavily on the choice of the step size ($\alpha$) and the communication matrix ($P$). Traditionally, these parameters require manual tuning or expert intuition. This paper addresses these limitations by proposing a "Learning-to-Optimize" (L2O) framework that automates hyperparameter tuning using Graph Neural Networks (GNNs).

Methodology: The GNN-ADMM Correspondence

The core innovation of this work is the mathematical realization of a "one-to-one correspondence" between distributed ADMM iterations and the message-passing framework of GNNs. A single iteration of distributed ADMM can be exactly represented as two distinct message-passing steps within a GNN architecture.

The diagram illustrates how ADMM iterations map to GNN message-passing (msg), aggregation, and update (upd) steps.

By exploiting this structural alignment, the authors treat $K$ iterations of ADMM as layers in a deep neural network, a process known as algorithm unrolling. Within this framework, Multi-Layer Perceptrons (MLPs) are used to predict learnable components at different granularities:

  • Graph-level: A global step size $\alpha$.
  • Node-level: Individualized step sizes $\alpha_i$ for each agent, allowing the algorithm to adapt to local objective difficulty.
  • Edge-level: Weighted communication links ($e_{ij}$), which effectively learn a problem-specific preconditioner for the network topology.

The model is implemented using JAX and the jraph library, enabling end-to-end training to minimize the error of the final iterate ($x^K$) relative to the true global minimizer ($x^\star$).

Experimental Setup

The framework was evaluated on two benchmark distributed optimization problems:

  1. Network Average Consensus: Agents collaborate to find the mean value of their private local data.
  2. Distributed Least-Squares: A collaborative linear regression task where each node possesses a subset of the total observations.

Experiments utilized synthetic problem instances generated on Erdős-Rényi random graphs with 8 nodes. The dataset comprised 900 training instances, with 100 instances each for validation and testing.

Results and Discussion

The experimental results demonstrate that the "Combined" model—which integrates learned node-level step sizes and edge weights—significantly outperforms standard baselines.

Performance comparison showing the error and consensus metrics for different ADMM variants across various iteration counts (k).

As shown in the table above, for the Consensus problem at the training target ($K=10$), the learned model reduced the error from 9.17 (baseline) to 1.96, representing a ~78% improvement. Similar gains were observed in the Least-Squares problem, where error dropped from 53.23 to 18.24. Notably, the learned variants achieved lower error levels in just 5 iterations than the baseline could reach in 10, indicating a substantial acceleration in convergence speed.

Furthermore, the model exhibited strong generalization. Although trained on $K=10$ iterations, the learned parameters continued to outperform the baseline when extended to $K=20$ iterations. This suggests that the GNN captures fundamental structural properties of the optimization process rather than merely over-fitting to a specific iteration count.

Conclusion and Future Work

This work demonstrates that GNNs are the natural architecture for distributed ADMM. By learning problem-specific hyperparameters, the proposed method significantly accelerates convergence and improves solution quality. While the current approach is memory-intensive due to unrolling, future work may explore implicit differentiation to scale to larger problems. Additionally, integrating communication compression or neural approximations of the local update steps could further enhance the efficiency and applicability of this framework in resource-constrained distributed environments.

Summary updated

Abstract

Distributed optimization is fundamental in large-scale machine learning and control applications. Among existing methods, the Alternating Direction Method of Multipliers (ADMM) has gained popularity due to its strong convergence guarantees and suitability for decentralized computation. However, ADMM often suffers from slow convergence and sensitivity to hyperparameter choices. In this work, we show that distributed ADMM iterations can be naturally represented within the message-passing framework of graph neural networks (GNNs). Building on this connection, we propose to learn adaptive step sizes and communication weights by a graph neural network that predicts the hyperparameters based on the iterates. By unrolling ADMM for a fixed number of iterations, we train the network parameters end-to-end to minimize the final iterates error for a given problem class, while preserving the algorithm's convergence properties. Numerical experiments demonstrate that our learned variant consistently improves convergence speed and solution quality compared to standard ADMM. The code is available at https://github.com/paulhausner/learning-distributed-admm.
BibTeX
@misc{doerks2025learning,
  author = {Henri Doerks and Paul Häusner and Daniel Hernández Escobar and Jens Sjölund},
  title = {{Learning to accelerate distributed ADMM using graph neural networks}},
  year = {2025},
  eprint = {2509.05288},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.05288},
}
Figure