Understanding Communication Backends in Cross-Silo Federated Learning

Amir Ziashahabi, Chaoyang He, Salman Avestimehr

TL;DR

Cross-silo federated learning requires moving massive machine learning models across highly diverse network environments, but standard communication backends like gRPC and MPI face severe trade-offs in speed or memory. This paper benchmarks these backends across LAN, geo-proximal, and geo-distributed setups, and introduces gRPC+S3—a hybrid backend that offloads heavy model payloads to cloud storage, achieving up to a 3.8× end-to-end speedup and massive memory savings over standard gRPC in wide-area networks.

What Problem Does This Paper Address?

Cross-silo federated learning (FL) allows different organizations (like hospitals or banks) to collaboratively train machine learning models without sharing their raw, private data. Because these silos can be located anywhere from the same server room to different continents, the underlying network conditions vary wildly.

With the rise of large-scale models like Large Language Models (LLMs) and Vision Transformers (ViTs), the choice of the communication backend—the software layer responsible for moving model updates between the server and clients—becomes a massive performance bottleneck. Existing options all have fatal flaws depending on the setup: MPI assumes a static, tightly managed network; PyTorch RPC is framework-locked; and standard gRPC underutilizes bandwidth on high-latency links because it multiplexes traffic over a single connection. While opening multiple gRPC channels can recover throughput, it causes the server's memory usage to skyrocket linearly with the number of clients, making large-model training impractical.

Fig. 1: Three common deployment environments for federated learning: (1) LAN: participants in a single building; (2) Geo- Proximal: participants in close geographical proximity; (3) Geo-Distributed: participants in distant locations. Context- aware selection of communication backends is crucial for optimizing performance in each setting.

Methods

The authors systematically evaluate three standard backends (MPI, gRPC, PyTorch RPC) and introduce a novel hybrid backend called gRPC+S3.

The core idea of gRPC+S3 is to decouple control from data:

  1. Sender Side: When a model needs to be sent, the backend splits the message. It serializes and uploads the heavy model weights to an Amazon S3 bucket. It then constructs a lightweight gRPC message containing only metadata and the S3 object key.
  2. Receiver Side: The receiver gets the small gRPC metadata message, extracts the S3 key, and fetches the actual model weights directly from S3.

By shifting the heavy data path to S3, receivers can pull data in parallel using independent, multi-connection downloads. This completely bypasses gRPC's single-connection bottleneck and eliminates the need for the server to buffer separate copies of the model for each client during a broadcast.

Fig. 3: Architecture of the proposed gRPC+S3 backend.

Datasets

The benchmarking suite evaluates both image and text tasks across four model size tiers to represent realistic FL workloads:

  • Small: ResNet56 (2.39 MB) trained on the Google Landmarks (GLD-23K) dataset.
  • Medium: MobileNetV3 (19.85 MB) trained on GLD-23K.
  • Big: DistilBERT (253.19 MB) trained on the 20 Newsgroups dataset.
  • Large: ViT-Large (1,243.14 MB) trained on GLD-23K.

Key Insights and Ideas

  • No Silver Bullet: The optimal backend is entirely context-dependent. In low-latency networks (LAN), CPU overhead (like serialization) is the primary bottleneck. In high-latency networks (WAN), efficient connection concurrency dictates performance.
  • The gRPC Memory Trap: Standard gRPC's lack of built-in multi-connection transfers severely hurts WAN performance. Fixing this by increasing concurrent dispatches causes server memory to scale linearly because each send buffers its own copy of the model.
  • Storage as a Communication Fabric: Treating cloud object storage (like S3) as a shared communication layer effectively converts a 1-to-N server broadcast into a 1-to-1 upload followed by parallel N-to-1 client downloads, shifting the scaling burden away from the central FL server.
Fig. 2: Effect of concurrent dispatch on gRPC: band- width (top) and memory (bottom) for North California →Bahrain.

Key Results

  • Geo-Distributed Speedups: In high-latency WAN environments, gRPC+S3 delivers a 3.5× to 3.8× end-to-end speedup over standard gRPC for large models.
  • Memory Efficiency: As shown in the paper's microbenchmarks, gRPC+S3 maintains constant server memory during concurrent broadcasts regardless of the number of clients, while gRPC and MPI memory usage scales linearly up to tens of gigabytes.
  • LAN Bottlenecks: In LAN environments, backends that avoid serialization (like PyTorch RPC and MPI with memory buffers) dominate. Standard gRPC is up to 9× slower for large models in a LAN setup because serialization accounts for up to 86% of its total latency.
Fig. 4: Peer-to-peer results across backends, environments, and model sizes. (a) CPU-to-CPU latency (log scale). (b) Speedup of concurrent over sequential transmission for 10 messages (Large uses 5). (c) Peak sender memory during concurrent broadcast.

Limitations and Future Work

  • Small Model Overhead: gRPC+S3 introduces a two-step transfer overhead (uploading to S3, then notifying via gRPC). For small models (<10 MB) or low-latency LAN environments, this round-trip overhead makes it slower than pure gRPC or MPI.
  • Cloud Dependency: The hybrid approach relies heavily on the availability and proximity of cloud object storage, which might conflict with strict data sovereignty rules or localized cross-silo setups that lack cloud integration.
  • Future Directions: The authors suggest that a robust real-world FL deployment should feature a versatile backend capable of falling back to pure gRPC for small payloads while dynamically engaging S3 for larger model updates.

Summary updated

Abstract

Federated learning (FL) has emerged as a practical means for privacy-preserving distributed machine learning. FL's versatile design makes it suitable for various training settings, from IoT edge devices in cross-device FL to powerful servers in cross-silo FL. A key consequence of this versatility is the high level of diversity found in the networking configuration of FL applications. Coupled with the rising demand for large-scale models such as large language models, well-informed selection and configuration of communication backends become crucial for ensuring optimal performance in FL systems. This work focuses on cross-silo federated learning, presenting in-depth benchmarks of various communication backends, including MPI, gRPC, and PyTorch RPC. In addition, we introduce gRPC+S3, a hybrid backend designed to overcome the limitations of existing approaches, particularly for transmitting large models across geo-distributed deployments, achieving up to $3.8\times$ end-to-end speedup over gRPC. Our benchmarks examine point-to-point and end-to-end performance for a broad range of model sizes running under realistic network conditions. Our findings provide practical insights for selecting and configuring suitable communication backends tailored to the specific federated learning tasks and network configurations.
BibTeX
@inproceedings{ziashahabi2026understanding,
  author = {Amir Ziashahabi and Chaoyang He and Salman Avestimehr},
  title = {{Understanding Communication Backends in Cross-Silo Federated Learning}},
  booktitle = {ICC},
  year = {2026},
  doi = {10.1109/icc59461.2026.11587537},
}
Figure