Peer-to-Peer Learning Dynamics of Wide Neural Networks

Shreyas Chaudhari, Srinivasa Pranav, Emile Anand, José M. F. Moura

This paper, presented at ICASSP 2025, provides a mathematical bridge between decentralized learning and deep learning theory. By leveraging the Neural Tangent Kernel (NTK), the authors transform the "black box" of peer-to-peer (P2P) neural network training into a predictable, linear dynamical system.

TL;DR

The authors provide an analytical framework to predict the training behavior of wide neural networks in decentralized, peer-to-peer environments. By linearizing the network dynamics using Neural Tangent Kernel (NTK) theory, they can accurately calculate how model parameters and loss will evolve over time without needing to run computationally expensive simulations.

What Problem Does This Paper Address?

Training deep neural networks in a peer-to-peer (P2P) fashion—where devices like smartphones or IoT sensors collaborate without a central server—is essential for privacy and "beyond-5G" edge computing. However, these systems are notoriously difficult to tune.

Because neural networks are highly non-convex, predicting how a specific learning rate or network topology (like a star vs. a cycle graph) will affect training usually requires "trial and error" through exhaustive simulations. This is resource-intensive and slow. The authors address the lack of a theoretical "crystal ball" that can tell researchers how a decentralized training run will perform before they actually hit "start."

Methods

The researchers combine two previously separate fields: Distributed Optimization and Neural Tangent Kernel (NTK) theory.

  1. NTK Linearization: The core idea of NTK is that as a neural network becomes very wide (overparameterized), its training dynamics simplify. It begins to behave like a linear model where the "kernel" (a matrix describing how gradients interact) stays constant throughout training.
  2. Gradient Flow Modeling: Instead of looking at discrete training steps, the authors model training as a continuous "flow." They derive closed-form differential equations for three popular P2P algorithms:
  3. Distributed Gradient Descent (DGD): Agents average their parameters, then take a gradient step.
  4. Adapt-Then-Combine (ATC): Agents take a gradient step, then average parameters.
  5. Combine-Then-Adapt (CTA): A variation of the diffusion-based approach.
  6. Analytical Solutions: For Mean Squared Error (MSE) loss, they derive a state transition matrix that allows them to calculate the exact state of the model at any time $t$ based only on the initial conditions and the network's communication graph.

Datasets

The authors validated their theoretical predictions using two primary setups:

  • MNIST (Subset): Used for testing affine (linear) models, specifically focusing on classifying the digits '0' and '1'.
  • Half Moons Dataset: A classic non-linear synthetic dataset used to test the dynamics of actual neural networks (specifically a single hidden layer with 256 units and sigmoid activation).

Key Insights and Ideas

  • The "Wide" Advantage: The most significant insight is that the "messiness" of non-convex optimization disappears in the infinite-width limit. This allows decentralized learning to be treated as a system of linear differential equations.
  • Predictability Across Topologies: The framework is robust across different communication structures. Whether the agents are arranged in a circle, a star, or a fully connected mesh, the analytical formulas accurately track the "error decay" during training.
  • Stability Guarantees: The paper provides a formal proof (Proposition 1) that DGD training in this regime is Bounded-Input Bounded-Output (BIBO) stable, provided the communication graph is doubly stochastic and the system is "minimal."
  • Hyperparameter Tuning: This work suggests that we can eventually optimize communication costs and learning rates mathematically rather than through heuristic testing.

Key Results

  • High Fidelity Predictions: In experiments with both affine models and neural networks, the "Predicted" loss curves (from the math) and the "Observed" loss curves (from actual training) were nearly identical.
  • Convergence Tracking: The formulas successfully captured how different agents in the network—starting with different local data—eventually reach consensus and minimize the global objective.
  • Validation of NTK in P2P: The results empirically confirm that NTK theory, which was developed for centralized training, holds up remarkably well in decentralized, multi-agent settings.

Limitations and Future Work

  • The "Wide" Requirement: The theory relies on the network being "sufficiently wide." While 256 neurons worked well for the Half Moons task, very narrow or highly compressed models might not follow these linear dynamics as closely.
  • Loss Function Constraints: While the paper provides closed-form solutions for MSE loss, other common functions like Cross-Entropy require numerical integration rather than a simple formula.
  • Static Topologies: The current work assumes the communication graph is static. In real-world wireless P2P settings, links often drop or change. Extending this to time-varying graphs is a logical next step.
  • Communication Efficiency: The paper focuses on characterizing dynamics; the authors suggest future work could use these characterizations to actively reduce the amount of data agents need to swap.

Summary updated

BibTeX
@inproceedings{chaudhari2025peer,
  author = {Shreyas Chaudhari and Srinivasa Pranav and Emile Anand and José M. F. Moura},
  title = {{Peer-to-Peer Learning Dynamics of Wide Neural Networks}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10890126},
}
Figure