Retrospective Remedy Steps for Demographically Fair Asynchronous Federated Training

When hospitals train a shared model without pooling data, the fast sites finish first. A server that waits for everyone pays in latency. A server that applies updates as they arrive, and discounts late ones as "stale," quietly lets the fast sites steer the model. Fast sites are often large, well-resourced centers with their own patient mix. The result is a demographic bias that comes from timing alone: the final model reflects who reported first, not who the patients are. Epidemiologists met the same confusion decades ago. A crude death rate rises simply because a population aged. They fixed it with direct standardization, which recomputes a rate as if every population had the same reference age mix (Wang et al., Yi et al.).

This note proposes a server rule borrowed from wireless communication. The server applies updates immediately, keeps a log of what it applied, and issues a later "remedy step" once slow sites report. The remedy makes the net update over each time window equal to a demographically standardized aggregate, and it also removes the part of the early updates that worked against the late-reporting groups. The move mirrors retrospective interference alignment. There, a transmitter sends blindly, learns afterwards (from outdated channel measurements) what interference each receiver suffered, and sends one correction that cancels it (Completely Stale Transmitter Channel State Information is Still Very Useful). The claim is that this recovers most of the fairness and accuracy of waiting for everyone, at close to the wall-clock speed of fully asynchronous training.

Core Claim

Two things are new here. First, we treat staleness in federated training as a demographic confound, which standardization can remove, rather than as noise to be discounted. Second, we correct the confound after the fact, rather than by waiting or by down-weighting late sites.

Let the demographic strata $s$ be pooled over sites, for example age band crossed with sex or race. Let $w_s$ be fixed reference-population weights. Let $\hat g_s(\theta_{t_s})$ be the gradient of stratum $s$, computed at the possibly old parameters $\theta_{t_s}$ that its site last received. Within a window $W$, the standardized target update is

$$\bar g_W = \sum_s w_s\, \hat g_s(\theta_{t_s}).$$

The server has meanwhile applied the cumulative update $A_W$, built only from whatever arrived in time. Once the remaining strata report, it applies the remedy

$$r_W = \lambda\,\big(\bar g_W - A_W\big) - \mu \sum_{s\,\text{late}} \Pi_{s}^{-}(A_W).$$

Here $\Pi_s^{-}(A_W)$ is the component of the applied update that points against late stratum $s$'s buffered gradient, that is, the negative projection onto $\hat g_s$ in the style of Gradient Surgery for Multi-Task Learning. The terms $\lambda, \mu \in [0,1]$ are shrinkage factors that grow with a per-stratum estimate of gradient noise. The first term restores the standardized composition. The second is the "retrospective cancellation": it undoes early damage that plain reweighting cannot see, because that damage is already baked into the parameters.

How the Directions Fit

The three directions play distinct roles.

Retrospective alignment supplies the protocol shape. The protocol has three stages: act on incomplete information, receive delayed side information, then broadcast one correction that is useful to all parties.

Our analog is to keep the log of stale gradients instead of discounting them.

Feedback-driven cancellation supplies the key trick. A party can subtract what it knows it contributed.

This literature also constrains the design. Do not Interfere but Cooperate notes that learned feedback codes assume perfect, noise-free feedback. Tandon et al. study asymmetric feedback. Stale gradients are both noisy and computed at old parameters, so exact subtraction is unavailable, which is why the shrinkage factors $\lambda, \mu$ are needed. We borrow structure from this literature, not theorems.

Demographic normalization supplies the target and the evaluation lens. It says what "correct" aggregation means: the reference-population weights $w_s$, not sample share and not arrival order.

It is also the part most likely to be dismissed, so it carries the main scientific question. Is timing bias real, and is standardization enough?

  • Fairness Guarantees under Demographic Shift (Shifty) motivates this. It shows fairness constraints fail at rates up to 100% under demographic shift, and it guarantees fairness only offline, by importance weighting plus confidence bounds. It needs more than 30k samples to avoid returning "No Solution Found."
  • We act during training, where each stratum's data is small and arrives late. That is exactly Shifty's weak regime.
  • FedHD aligns feature statistics across heterogeneous centers once, via distillation. It has no notion of timing, arrival order, or correcting past updates.

Research Program

The work proceeds in three steps.

Step 1: measure timing-induced bias. We simulate asynchronous federated training in which each site's reporting delay is correlated with its demographic composition. We then measure how the stratum-standardized loss and the worst-stratum loss drift relative to synchronous training. Delay is the controlled variable, swept from zero to several local epochs. If the bias does not appear, the premise fails, and we will report that.

Step 2: build and analyze the remedy rule.

  • The server keeps a bounded buffer of per-stratum gradients.
  • It chooses $\lambda$ and $\mu$ by a simple signal-to-noise estimate from each stratum's minibatch variance. This answers the concern raised in Gap 4 of the research file about exact subtraction versus finite-sample noise.
  • We derive a first-order bound. The remedy exactly reproduces the standardized update when the loss is locally quadratic and the delays are shorter than the curvature timescale. Its error grows with delay times the curvature norm. This gives a predicted failure point we can check empirically, because the remedy is a linearization around old parameters.

Step 3: test whether the cancellation term matters beyond reweighting. This is the question that decides whether the idea is more than bookkeeping.

Baselines

Synchronous FedAvg with standardized weights (the naive reference). The server waits for all sites every round and weights updates by $w_s$. It is the accuracy and fairness ceiling for this model and data. It is the "orthogonalize and wait" option: slow but clean. We run it with the same local steps, learning rate grid, and total gradient evaluations as the other methods, and report wall-clock under the same simulated delay profile. Beating it means matching its standardized AUC within its seed-to-seed spread while using substantially less wall-clock time. No reported result was found on Lacuna for federated CheXpert or CheXpert Plus training under this protocol.

Asynchronous training with staleness discounting. Each update is applied on arrival, scaled down by its age, and weighted by sample share. This is the default the method must beat on fairness. No reported result was found on Lacuna for any staleness-discounted asynchronous method on these datasets, so we implement it ourselves under the same tuning budget.

Asynchronous training with standardized weights but no remedy. Arriving updates are weighted by $w_s$ at application time. This is the strongest simple competitor and the key ablation. It isolates whether the retrospective step, and specifically the $\mu$ cancellation term, adds anything.

Asynchronous training with PCGrad on buffered stratum gradients. We apply the projection rule of Gradient Surgery for Multi-Task Learning whenever gradients conflict, using the latest buffered gradient of each stratum, with no standardization. It is the strongest existing method for conflicting gradients. PCGrad reports raising Meta-World MT50 success from under 40% to about 70%, and CIFAR-100 accuracy from 67.7% to 77.5% with routing networks. Neither result involves delay or demographics.

Our method. We report it at $\mu = 0$ and at tuned $\mu$, which yields the second ablation.

External reference points, not like-for-like baselines.

  • The centralized CheXpert model reports AUC 0.97 for Pleural Effusion and 0.94 for Edema (CheXpert). It serves as a pooled-data upper reference.
  • FedHD reports 91.2% accuracy on CAMELYON16 (FedHD). It is a federated histopathology reference.
  • FedHD's summary gives no CAMELYON17 number, so no reported result was found on Lacuna for CAMELYON17 accuracy.

Why not just…?

Why not just wait for all sites each round? Waiting removes timing bias entirely. It is the orthogonal, discard-and-retransmit option that retrospective alignment was designed to beat (Completely Stale Transmitter Channel State Information is Still Very Useful). Its cost is wall-clock time bounded by the slowest site. We report the accuracy versus wall-clock frontier. If the synchronous baseline sits on that frontier at realistic delay ratios, the proposal has no practical point.

Why not just reweight arriving updates by reference-population weights? This is the real threat. To first order, reweighting and remedying agree whenever every stratum reports within the window and curvature is low. The remedy can only win in two cases: when late strata's missing mass is large, or when early updates have already pushed the parameters against late strata. That is exactly what the $\mu = 0$ versus tuned-$\mu$ comparison and the delay sweep test. If standardized reweighting matches the remedy across the sweep, we will conclude that standardization is the whole contribution, and report the timing-bias finding on its own.

Why not just apply PCGrad to the stale gradients? PCGrad removes pairwise conflict but ignores population weights. In the terms of Gradient Surgery for Multi-Task Learning, it leaves the "magnitude imbalance" leg of its own triad to projection alone, so over-represented strata still dominate. Its projections are also recomputed on each arrival rather than settled once per window. The comparison on worst-stratum AUC shows whether this matters.

Why not just train separate per-site models? Separation discards transfer across sites. PCGrad's Meta-World results show jointly trained agents beating independent single-task agents. Small sites, which are the slow ones here, lose the most from separation. We include per-site models as a sanity row.

Proposed Experiments

Main testbed: CheXpert Plus. We use CheXpert Plus, which provides age, sex, race, and insurance type for 64,725 patients. We partition patients into 8 to 16 simulated sites with skewed demographic mixes. Each site's delay is tied to its size and composition, and we also run a control with delay independent of demographics. Strata are age band crossed with sex, with race as a secondary axis. The reference weights $w_s$ are the pooled cohort's composition. Evaluation uses the consensus-labeled CheXpert test set.

Metrics:

  • per-pathology AUC, directly standardized over strata;
  • worst-stratum AUC;
  • the gap between the best and worst stratum;
  • wall-clock time to a target standardized AUC.

Real-site heterogeneity: CAMELYON17. Its five real centers carry scanner heterogeneity rather than demographic heterogeneity. This tests whether the mechanism generalizes to site strata.

Shift after training: Adult and UFRGS. Following Shifty's protocol, we train on Adult and UFRGS split into delayed clients, then apply Shifty's demographic shift at deployment. This asks whether standardized training reduces the fairness failure rate that Shifty reports for conventional methods, with Shifty itself as a post-hoc filter.

Decisive checks. Four results decide the outcome:

  • whether timing bias appears when delay correlates with demographics and vanishes in the independent-delay control;
  • whether tuned $\mu$ beats $\mu = 0$ on worst-stratum AUC at moderate delays;
  • whether that advantage shrinks, as the first-order bound predicts, once delay exceeds the curvature timescale;
  • whether noise-aware shrinkage prevents degradation when strata are small, where exact subtraction should amplify noise.

If the second check fails, the idea reduces to standardized asynchronous averaging, and we will say so.