Wall-Clock Break-Even for Federated Learning by Synthetic Data Sharing

Federated learning trains one model across many data holders without moving their raw data. A recent shortcut lets each client compress its private data into a small synthetic dataset and send that once, instead of exchanging model weights over many rounds. The headline evidence for the shortcut counts bytes: on CIFAR-10 with ResNet-18, sending synthetic images took about 35 MB, against about 8.7 GB for twenty rounds of weight averaging. Bytes are not time, though. Each client first has to run its own synthesis, an iterative optimization of synthetic images, and the slowest, farthest client still sets when the server can start training. We propose to measure, model, and map the conditions under which sharing synthetic data actually finishes sooner than weight averaging. Here "finishes sooner" means reaching a target accuracy in less wall-clock time on a realistic wide-area network with heterogeneous client hardware.

The deliverable is a break-even frontier. It is drawn over client compute speed, network round-trip time and bandwidth, and how unevenly data is split across clients, and it is backed by an interpretable cost model checked against network emulation. A second question follows directly from the first. If a server enforces a deadline, clients that are slow or far away are the ones whose synthetic data never arrives. We test whether that systematically erases the data distributions of distant regions.

Core Claim

The claimed communication advantage of aggregation-free federated learning is a statement about byte volume. Its wall-clock advantage depends on a different and unmeasured quantity: the time each client spends on local synthesis relative to its network transfer time. We expect a sizable region of realistic settings where weight averaging reaches a given accuracy first. This region should appear when clients are compute-poor but well connected, or when data is only mildly uneven so that weight averaging converges in few rounds. We also expect the region where synthetic sharing wins to be larger than byte counts suggest on high-latency links, because one shot avoids paying the round-trip time many times over. No paper in our file reports wall-clock time for either regime. We therefore treat the claim as an empirical hypothesis that the experiments can refute in either direction.

Positioning

The method under test is FedUSD. In FedUSD each client distills its local features into synthetic images by matching their dominant singular directions (via SVD) and their variance (via Earth Mover's Distance). The server then trains on the pooled images. FedUSD motivates this work through two reported results: the 30.82% accuracy gain on SVHN at Dirichlet α = 0.01, and the 35 MB versus 8.7 GB transfer comparison. Its evaluation runs on simulated partitions and reports neither client synthesis time nor wall-clock time. We do not change FedUSD's algorithm. We change how it is judged.

Direction 1 supplies both the warning and the tools. Earlier work found that proxy metrics such as communication volume or round counts misrepresent wall-clock behavior (Reddi et al.; Xu et al.), and that communication costs can erase the gains a distributed algorithm promises on paper (Giachin and Rullent). We borrow two tools:

  • Emulator. Analyzing Geospatial Distribution in Blockchains runs real protocol code under Linux traffic shaping (netem), using measured ping delays between 242 cities. That study showed that geographic latency alone drives distant nodes to 0% liveliness.
  • Cost model. DP-perf predicts runtime by counting arithmetic operations and memory traffic per operator and adding a simple message-latency model (LogP). It stays under 20% error on unseen machines.

The closest system-level precedent is FedHPO-Bench. It adds an analytical model of client compute, bandwidth, and connection latency to federated hyperparameter search, but it covers only weight-averaging training and never generates synthetic data. CaPS generates synthetic data collaboratively across data holders and leaves low-bandwidth operation open. Its cryptographic protocol is a different cost structure, and we do not study it here.

Research Program

The first component is an instrumented testbed. Real FedUSD clients and real FedAvg clients run in containers. Pairwise delays are drawn from the 242-city latency matrix used by GeoDec, and netem additionally imposes bandwidth caps, jitter, and asymmetric uplinks. Client compute heterogeneity is created in two ways: CPU and GPU throttling, and a mix of device classes. Every run logs per-client synthesis time, serialization time, transfer time, and server training time. It also logs the wall-clock time at which test accuracy first crosses fixed targets.

The second component is an interpretable cost model in the style of DP-perf. Synthesis time is predicted from operation and memory counts for the SVD, the distance-matching, and the augmentation steps, measured once per device class. Network time is predicted from latency, bandwidth, and message size. For synthetic sharing, completion time is the slowest client's synthesis-plus-upload time, plus server training time. For weight averaging, it is the sum over rounds of each round's slowest client. The model's purpose is to produce the break-even frontier cheaply across settings too numerous to emulate. Emulation checks the model at sampled points.

The third component is the deadline study. Under a server cutoff, we record which clients' synthetic data is lost and how that maps to geography and to label distribution. We then measure per-class and per-client-group accuracy of the resulting global model.

Baselines

The naive reference point is the existing byte-count comparison. Under it, FedUSD is preferred whenever it transfers fewer bytes. We report how often this rule picks the method that actually reaches target accuracy first in emulation. If the rule is right nearly always, our main hypothesis fails.

The strongest competing method is FedAvg as configured in FedUSD's own comparison. It runs on the same CIFAR-10, CIFAR-100, and SVHN partitions at α ∈ {0.01, 0.05, 0.1}, with the same model architectures. Both methods get the same tuning budget, a fixed number of configurations each. The reported reference numbers are FedUSD's 35 MB versus FedAvg's 8.7 GB on CIFAR-10/ResNet-18 and its 30.82% SVHN gain at α = 0.01. No reported wall-clock result for either method was found on Lacuna. The claim that "synthetic sharing wins" counts only if FedUSD reaches the matched target accuracy first in emulated wall-clock time. The claim that "FedAvg wins" counts by the same criterion in reverse. The finding of interest is any setting where the winner differs from the byte-count prediction.

The strongest competing estimator is FedHPO-Bench's style of analytical system model: round-trip time plus size over bandwidth, with compute time drawn from a static per-client estimate. It is the cheaper alternative to our cost model and to emulation.

The ablation that isolates our key ingredient is our cost model with the synthesis-compute term removed. This variant is exactly "bytes plus latency." If it locates the frontier as accurately as the full model, then synthesis compute is not what matters, and our central claim is wrong.

Why not just…?

Why not just use a closed-form latency formula instead of emulation? This is a real threat. For a single bulk upload, jitter and queueing may matter little, and the formula may be adequate. Settling that is the point of comparing the formula against emulation. Our operating rule: if the formula's prediction of which method finishes first agrees with emulation across the sampled grid, the formula becomes the recommended tool and emulation is kept only for validation. We expect disagreement mainly in the many-round FedAvg regime. There, the GeoDec results show that per-round waiting on distant nodes compounds, and Parsimon (Scalable Tail Latency Estimation) shows that tail latency is governed by congestion dynamics a static formula omits.

Why not just add a FLOP count to the byte count? Counting FLOPs is part of our cost model, so the real question is whether FLOPs alone are enough. DP-perf found that memory traffic, not arithmetic, dominates some operators. We test FLOP-only, memory-aware, and full models against measured synthesis time, and we keep the simplest one that ranks methods correctly.

Why not just run on real geo-distributed cloud regions? Real deployments are the ground truth, but they are costly, uncontrolled, and hard to reproduce. We use a small real cross-region deployment, a few regions and a few runs, only to calibrate how far emulation diverges from reality. All sweeps run under emulation.

Proposed Experiments

The first decisive check is a ranking test. On the full grid of client compute classes, latency regions, bandwidths, and α values, we measure how often the byte-count rule, the formula, and our cost model each pick the method that emulation shows reaching 80% and 95% of FedUSD's final accuracy first. Success means two things: the byte-count rule misranks a substantial share of realistic settings, and the cost model's runtime error stays near DP-perf's under-20% level when checked against emulation.

The second check traces the break-even frontier in client speed versus round-trip time, separately for each α. This shows whether FedUSD's accuracy advantage under extreme data imbalance survives once its synthesis time is charged to it.

The third check is the deadline study. Using geography-heavy layouts that mirror GeoDec's majority-in-US/Europe placement, we impose cutoffs and measure two things. One is the fraction of high-latency clients whose synthetic data is dropped. The other is the accuracy gap on classes concentrated in those clients. A positive result means latency reproduces, in aggregation-free learning, the centralization effect GeoDec found in consensus protocols. A null result means one-shot sharing is robust to it, which would itself be an argument in FedUSD's favor.