Communicating When Synthetic Noise Disagrees

Many distributed training jobs now run largely on generated data. Gravitational-wave detectors are a clear example: these instruments see real signals so rarely that classifiers learn almost entirely from simulated chirps injected into recorded detector noise. Other jobs rebalance rare classes by synthesizing minority examples. Some fraction of these synthetic samples is wrong: mislabeled, physically impossible, or placed inside the wrong class's region of feature space. Separately, distributed training saves bandwidth by letting each machine (a "worker") take several local optimization steps between synchronizations. The best-known adaptive rules choose that interval from the training loss or the learning rate. Neither rule asks what the data is made of. This proposal argues that the two choices interact, and that the loss-driven rule fails in a specific, predictable way. A mislabeled synthetic sample can never be fit properly, so it keeps the training loss high. A loss-driven scheduler therefore keeps synchronizing rarely at exactly the moments when workers are drifting apart by memorizing their own private synthetic errors.

The proposed fix uses a cheap signal that distributed training already almost has. When each worker generates its own synthetic samples with its own random seed or its own local oversampler, the errors in those samples are independent across workers, while the useful content is shared. This is the same statistical fact that lets multi-messenger astronomy shrink a broad sky region by multiplying independent likelihood maps. LLAMA intersects gravitational-wave and neutrino probability maps because the two instruments' errors are uncorrelated. By analogy, at each synchronization we measure how much the workers' synthetic-data updates agree. That agreement score then decides two things: how long the next local phase should be, and how much weight each group of synthetic samples should receive. Raw data never needs to be exchanged.

Core Claim

Let $s_i$ be a low-dimensional random projection (a "sketch," here 1,024 numbers) of the parameter change that worker $i$ accumulated from synthetic samples alone during the last local phase. Let $n$ be the number of workers. We define the synthetic coherence

$$\rho = \frac{\left\| \frac{1}{n}\sum_{i=1}^{n} s_i \right\|^2}{\frac{1}{n}\sum_{i=1}^{n} \|s_i\|^2},$$

which is near 1 when workers' synthetic updates point the same way. It falls toward $1/n$ when they are dominated by independent errors, because averaging $n$ independent zero-mean vectors divides their squared norm by $n$. The same ratio computed per synthetic group $g$ (a class, a signal-to-noise bin, or a generator) gives $\rho_g$.

The claim has three parts. First, under worker-independent synthetic noise, the best communication interval depends on $\rho$, and the training loss is a poor proxy for $\rho$. Second, a scheduler driven by $\rho$, together with per-group down-weighting of incoherent synthetic groups, reaches a target accuracy with fewer bytes than loss-driven or round-driven schedules. Third, when the synthetic noise is shared across workers (the same flawed generator everywhere), $\rho$ cannot see it and the method should give no gain. That third prediction can be falsified and marks the method's boundary.

Relation to Prior Work

Two lines of work supply the scheduling machinery, and both ignore data provenance. ADACOMM shrinks the local interval in proportion to the square root of normalized training loss. Decaying local steps in FedAvg cuts the number of local steps by round count, by loss, or when validation accuracy plateaus. Both assume clean, genuine data. 0/1 Adam does the opposite and lengthens intervals as the learning rate decays. We do not try to resolve that larger tension in general. We test a narrower mechanism under which a loss-based signal is specifically misleading.

Three papers motivate the noise model. Local density-aware oversampling prunes noisy seeds and restrains interpolation toward the other class, but it needs a global nearest-neighbor graph on one machine. MixAnN learns where to synthesize with reinforcement learning and reports that classifier overconfidence on mislabeled synthetic points corrupts its reward signal. S³VM-OvS treats synthetic points as unlabeled to avoid label noise, but relies on quadratic-programming solvers. All three are centralized. Our per-group weighting is a distributed replacement that consumes only update sketches.

The gravitational-wave literature supplies the testbed and the physically defined corruption. MLGWSC-1 found that ML searches lose ground on real noise, partly because they fail to exploit consistency of arrival time and phase across detectors. That consistency is exactly what an unphysical synthetic injection breaks. Trovato et al. show that extreme imbalance at false-alarm rates near $10^{-6}$ requires computing the loss directly from logits. We adopt that practice. Gravity Spy glitch classifications document strongly site-specific glitch rates, which we use to build a realistic non-IID split of noise across workers.

Research Program

Phase 1: measurement study. We train with fixed intervals over a grid of interval lengths, synthetic noise fractions, and noise structures (worker-independent versus shared). We then ask whether the best interval moves with the noise fraction, and in which direction. One possibility is that long local phases compound private noise. The other is that averaging acts as a low-pass filter, as Gap 5 of the file poses. If the best interval does not move under independent noise, the scheduling half of the proposal is dead. Only the weighting half would remain worth pursuing, and we would report that.

Phase 2: estimator. Each worker keeps a separate running sum of updates from its synthetic minibatches. At each sync it sends a 1,024-dimensional sketch of that sum per group, alongside its normal update. This adds well under one percent of payload for the models considered. We validate that $\rho$ tracks the true corrupted fraction, using controlled runs where corruption labels are known.

Phase 3: rule. The next interval is ADACOMM's interval multiplied by $(\rho/\rho_{\text{ref}})^{\beta}$, clipped to a fixed range. The sign of $\beta$ comes from Phase 1, and its magnitude is tuned on a held-out configuration. Group weights are set proportional to how far $\rho_g$ sits above $1/n$.

Phase 4: shared-noise control. We run the shared-noise condition to confirm the predicted null result.

Testbeds. Testbed A is MLGWSC-1 Datasets 3 and 4. Each worker generates precessing binary-black-hole injections on the fly from its own seed. We add two controlled corruptions to a fraction $p$ of injections. The first shifts the inter-detector arrival time beyond the roughly 10 ms Hanford–Livingston light-travel time. The second labels injections below detectability as signals. For Dataset 4, background noise is split across workers by detector and observing period to create heterogeneity. Testbed B is CIFAR-10 with one class downsampled to 5%, following the MixAnN protocol, and split non-IID across 16 workers. Each worker oversamples its minority class with SMOTE on its own few examples. This produces the worker-specific boundary distortion that Gap 2 of the file describes, without any artificial label flipping.

Baselines

The naive reference point is synchronous data-parallel training: every worker communicates after every step, using the same synthetic data. It defines the accuracy ceiling for the given data and the communication cost of not adapting. The second baseline is local SGD with a fixed interval, swept over a grid. Each grid point gets the same tuning budget as the other methods, and we report the best one, so the "fixed" baseline is a strong one.

The strongest scheduling competitors are ADACOMM and the round-based and loss-based decaying local-step schedules. On clean data these report a 3.3× wall-clock speedup to target loss with VGG-16 on CIFAR-10 (ADACOMM) and an 18% validation-accuracy gain over fixed local steps on CIFAR-100 (decaying local steps). No reported result found on Lacuna for either method under synthetic sample noise. We also include a learning-rate-coupled expanding interval in the spirit of 0/1 Adam, without its 1-bit compression, so that only the scheduling rule differs.

The strongest data-side competitor is local denoising before training. Each worker applies the density-aware filter and restrained interpolation to its own data, then trains with ADACOMM. In the gravitational-wave case the analog is an SNR cutoff on injections. For detection, the external reference is MLGWSC-1's reported result that the leading ML search reached 70% of PyCBC's sensitive distance at one false alarm per month on real O3a noise. We use the challenge's own evaluation code.

Fairness. Every method gets the same synthetic sample stream, model (a 1-D InceptionTime-style network for testbed A, ResNet-18 for testbed B), total gradient-step budget, and twenty tuning trials. We report bytes communicated, simulated wall-clock time under a fixed bandwidth-and-latency model, and measured time on a 16-GPU cluster.

Success criterion. "Beating" a baseline means a statistically separated improvement over five seeds on one of two measures: fewer bytes to reach the best baseline's final metric, or a higher final metric at equal bytes. The metrics are sensitive distance at one false alarm per month for testbed A and minority-class F1 for testbed B.

Ablations. Four ablations isolate the key ingredient. (a) Coherence computed on the whole update rather than the synthetic part, which tests whether separating synthetic from real data matters. (b) Coherence driving only the interval. (c) Coherence driving only the group weights. (d) An oracle that knows which samples are corrupted and drops them, as an upper bound.

Why not just…?

Why not just filter the synthetic data first and use any standard schedule? This is the most serious threat. If local filtering removes most corrupted samples, the schedule never sees the noise, and the interaction we study disappears. The evidence cuts both ways. The density-aware filter works with a global neighbor graph, and its behavior on the handful of minority samples one worker holds is untested. Our unphysical time-delay corruption is invisible to any single-detector density test, yet coherence across workers can still detect it. The filter-plus-ADACOMM baseline settles the question directly. If it matches our method on both testbeds, we will report that the scheduling contribution is unnecessary.

Why not just use ADACOMM or decaying local steps as published? Our central hypothesis is that unfittable synthetic samples hold the training loss up, so a loss-driven rule keeps intervals long while drift grows. The round-based rule cannot react to noise at all. Phase 1 tests this directly by recording ADACOMM's chosen intervals against the noise fraction. If ADACOMM already shortens intervals appropriately, the hypothesis is wrong.

Why not just synchronize every step and avoid the interaction? Synchronous training is our naive reference. If it reaches the target within the same byte budget, local updates are not worth it in this regime. Photon reports 512× communication reduction from periodic averaging in LLM pre-training, which suggests local updates matter at scale. However, we treat the bandwidth model as a stated assumption, not as evidence for our specific case.

Why not just treat synthetic samples as unlabeled, as S³VM-OvS does? For testbed A this is ill-posed: injections are the only positives, so removing their labels removes the signal class. For testbed B, we include a consistency-regularized unlabeled-synthetic variant as an additional data-side baseline under the same budget.

Decisive Checks

The proposal stands or falls on three results. First, the best fixed interval must shift measurably with the independent-noise fraction while ADACOMM's chosen interval does not. Second, $\rho$ must correlate with the true corrupted fraction across configurations. Third, the gain must vanish under shared noise. If all three hold and the method beats filter-plus-ADACOMM in bytes-to-target on MLGWSC-1 Dataset 4, the claim is established. If only the weighting ablation carries the gain, we will report a distributed noise filter, not a scheduler.