Conflict-Aware Synchronization Schedules for Training Across Data Silos

Many organizations that want to train one shared model cannot pool their data. Hospitals, agencies and enterprise departments keep records behind their own firewalls and let only aggregates leave. The usual workaround is low-communication training: each site trains a copy of the model locally for many steps, and a coordinator periodically averages the copies. This cuts network traffic, but it averages every site with every other site on the same schedule. When sites hold very different data, their copies pull in conflicting directions, and averaging them can erase what each one learned. We propose that the coordinator measure, from messages it already receives, which sites agree and which conflict. It would then spend its communication budget accordingly: frequent averaging inside groups of sites that agree, rare and partial averaging between groups that conflict, and a final weighted merge.

The opportunity sits between two bodies of work. Federated data architectures, such as polystores, medical research networks and data meshes, already enforce a hub-and-spoke shape in which a central mediator talks to autonomous sites and only summaries cross institutional boundaries. Gradient-synchronization research, meanwhile, has made it cheap to train across slow links, but it assumes sites with similar data and symmetric peer-to-peer networks. Our claim is that under the federated constraint the most useful lever is not compressing each message further. It is deciding who synchronizes with whom and how often, and that decision can be made without moving any data beyond what low-communication training already moves.

Core Claim

In a hub-and-spoke federation where sites hold non-identically distributed data, a schedule that groups sites by the agreement of their recent model updates should reach better quality than uniform periodic averaging at the same number of bytes through the coordinator. It should do so without the centralized calibration pass that current conflict-aware methods need, and without requiring a starting checkpoint that is already "merge-ready", meaning located in a flat region of the loss surface where averaged copies stay good.

The closest existing work is MERIT. MERIT computes gradients on about 200 samples per task, runs principal component analysis on them, splits conflicting tasks across isolated clusters that never communicate, and merges the clusters' models once at the end with a token-weighted average. Our proposal differs in three ways that follow from the federated setting.

First, sites are fixed partitions. We cannot reassign data the way MERIT reassigns tasks, so we group sites rather than data.

Second, the conflict signal comes from updates the sites already send, not from a central gradient pass. That central pass is exactly what non-export rules in networks like HiGHmed forbid.

Third, groups are recomputed during training and keep exchanging a little information. MERIT's one-shot merge depends on a merge-ready checkpoint, and the paper itself says it degrades from raw base models.

Mechanism

The base procedure is DiLoCo, as characterized in the DiLoCo scaling-law paper. Each site runs $H$ local optimizer steps. The coordinator then treats the change in each site's weights as a "pseudo-gradient" and applies the average change through an outer optimizer, Nesterov SGD.

We keep this loop and add one step at each outer round $r$. Each site $i$ sends a short random projection $s_i^r$ of its pseudo-gradient, a few kilobytes, alongside the update itself. The coordinator scores pairwise agreement as

$$ a_{ij}^r = \cos\left(s_i^r, s_j^r\right), $$

and keeps an exponential moving average of these scores. A random projection approximately preserves cosine similarity, which is why the cheap sketch can stand in for the full update. The coordinator clusters sites on the smoothed agreement matrix into groups whose members tend to agree.

Inside a group, sites are averaged every $H$ steps as in DiLoCo. Across groups, the coordinator averages only every $\kappa H$ steps, and only on the parameter blocks where the groups' models diverge least in function. "Function" here means output behaviour: we measure divergence as the Jensen–Shannon distance between output distributions, following Factored Gossip DiLoCo. That paper found output-level divergence a better guide than raw weight distance, but it fixed its 43% parameter subset offline. We instead choose the subset per round under a fixed byte budget. At the end, group models are merged with MERIT's token-weighted average.

The asymmetric topology is deliberate. All reduction happens at the coordinator, which plays the role of a polystore mediator like the one in The Data Civilizer System. Sites therefore never need inbound peer connections or collective GPU primitives across administrative domains.

Research Program

We would build this as a simulator with a single coordinator and $N$ site processes. Every message passes through a metered channel that records bytes, enforces bandwidth caps, and injects stragglers and dropped sites. This makes communication cost exact and comparable across methods.

Three questions organize the analysis.

The first is whether conflict measured from sketches tracks conflict measured from full pseudo-gradients. If it does not, the mechanism has no reliable signal. We will report rank correlation between the sketch-based and full-update agreement matrices as a function of sketch size.

The second is whether agreement structure actually changes over training. If it does not, dynamic regrouping adds nothing over a grouping fixed at the first round.

The third is whether message compression interacts with heterogeneity. We would optionally compress pseudo-gradients with the frequency-domain top-$k$ scheme of DeMo, which transforms updates with a discrete cosine transform and keeps the largest coefficients. DeMo's assumption that most update energy sits in few low-frequency coefficients was established on homogeneous pretraining. We will measure how much energy the retained coefficients capture as sites become more heterogeneous.

On privacy, the agreement matrix exposes pairwise similarity between sites. That is a new piece of information, and attacks like label leakage in split learning show that intermediate signals can reveal private data. We will therefore report what the coordinator can infer about site label distributions from the agreement matrix alone. We will not claim formal privacy.

Baselines

The naive reference point is training each site's model in isolation with no communication. This bounds what sharing buys. Its counterpart with no communication constraint is joint training on pooled data. Pooling would be illegal in a real federation, but it is a useful upper reference. On Vision-FLAN with Qwen2.5-VL-3B, MERIT reports that centralized joint training reaches 54.3 on its 8-benchmark average.

The strongest communication-efficient baseline for the base-model regime is DiLoCo with uniform averaging at $H=30$, run through the same coordinator. Its scaling-law paper reports compute utilization above 80% at 10x to 100x less bandwidth than standard training, but the file has no reported validation loss for it. We will therefore run it ourselves at matched bytes. Factored Gossip DiLoCo is the strongest method that already synchronizes only part of the model. It reports compute utilization rising from 53% to between 66% and 100% at 200 Mbps. We adapt its blocking gossip step to the star topology, since sites cannot gossip directly through firewalls.

The strongest conflict-aware baseline is MERIT. On Vision-FLAN with Qwen2.5-VL-3B it reports a score of 57.0 on the 8-benchmark average (against 54.3 for joint training) and 52.0 on LLaVA-Wild (against 41.9). We run it in two forms. The first is as published, with its centralized calibration pass. The second is a governance-compatible version in which each site computes its 200-sample gradient locally and sends only a projection, followed by MERIT's static clustering and one-shot merge.

All methods get the same site partition, the same total training tokens, the same byte budget through the coordinator, and the same hyperparameter search of twelve configurations per method.

Two ablations isolate our ingredient. The first freezes the groups found at round one. The second assigns sites to groups at random but keeps our schedule and byte budget. Beating these means one of two things. In the merge-ready regime, it means matching MERIT's 57.0 without a central calibration pass. In the base-model regime, it means lower validation loss than uniform DiLoCo at equal bytes, with the gap shrinking under the random-grouping ablation.

Why not just…?

Why not just run DiLoCo with every site averaged together? This is the default, and on near-identically distributed data it is probably right; we expect no gain there and will show it. The open question is strong heterogeneity. MERIT shows that joint multi-task training suffers from conflicting gradients, producing short-answer collapse on LLaVA-Wild (41.9 against 52.0). WASH documents that copies trained in isolation drift into different basins of the loss surface. DiLoCo's scaling laws were measured only on C4 and Dolma used as a single homogeneous stream. Our source-partitioned experiment tests this alternative directly at matched bytes.

Why not just run MERIT with locally computed gradient sketches? This is the real threat, because it already satisfies the governance constraint. It differs from our method only in grouping once and merging once. If it matches us, the right conclusion is that dynamic scheduling is unnecessary, and we would report that. We expect it to hold up on merge-ready instruction tuning but to fail from base models, where MERIT's own analysis says one-shot averaging breaks. The frozen-grouping ablation and the base-model experiment settle the question.

Why not just compress harder and keep uniform averaging? Compression methods such as DeMo, BitDP and SDP4Bit reduce the bytes per message, and DeMo reports an 85x reduction on a 300M-parameter OLMo model. None of them changes what gets averaged with what, so the conflict between sites survives compression. They are complementary rather than competing, and we include DeMo-compressed versions of both our method and DiLoCo. The relevant constraint is that BitDP needs All-Gather and assumes a symmetric gradient distribution, neither of which is guaranteed in a federation.

Why not just use per-site personalized models? Personalization gives up the single shared model that federations such as the Medizininformatik-Initiative want. The isolated-training baseline measures how much sharing buys over it.

Proposed Experiments

The first experiment uses Vision-FLAN. We assign its source datasets to 16 simulated sites by task family, so that conflicts follow site boundaries, and fine-tune Qwen2.5-VL-3B with the 8-benchmark suite MERIT uses: SeedBench, MMBench, LLaVA-Wild, MMVet, TextVQA, AI2D, MathVista and MMMU. This is MERIT's home turf and tests whether we can match its 57.0 without central calibration.

The second experiment pretrains an OLMo-300M-style model from scratch on Dolma, with sites partitioned by source. Here no merge-ready checkpoint exists. We plot validation loss against cumulative coordinator bytes and against simulated wall-clock time at 200 Mbps, the bandwidth Factored Gossip DiLoCo used. We repeat the run with FineWeb split by crawl domain to check that conclusions do not depend on one corpus.

The decisive checks are four. Dynamic grouping must beat both frozen and random grouping in the base-model run. Sketch-based agreement must correlate with full-update agreement. Our method must hold its margin when 10% of sites drop out each round. And the coordinator's ability to infer site label distributions from the agreement matrix must stay near chance, or the governance claim weakens and we will say so.