Horizon-Diverse Replicas for Communication-Light Training of Sequence Forecasters

Forecasting models are usually trained to predict one step ahead and then run on their own outputs to produce long forecasts, and their errors pile up as they go. A known fix is to keep the one-step model but train it on a loss that unrolls it many steps into the future. This proposal makes a simple observation about that fix and then tests a bet built on it. The observation is that unrolling raises compute per update roughly in proportion to the horizon, while the gradient that workers must exchange stays the same size, because the parameter count does not change. Long-horizon training is therefore unusually cheap to distribute. The bet is that, when several workers train mostly independently and only occasionally average their weights, each worker should be given a different training horizon. Short-horizon workers follow a smooth but biased objective, long-horizon workers follow a rugged but less biased one, and periodic averaging combines them. We expect this to escape the low-dimensional "simple" solutions that recurrent models fall into, at a fraction of the communication of standard training.

Put in two sentences: give each worker in a periodically synchronized cluster its own forecasting horizon, so that the workers' disagreement explores the space in a structured way rather than through random noise. The test is whether this beats the obvious alternative of giving every worker the same mix of horizons, at equal compute and equal bytes sent, on long-horizon accuracy and on how rich the learned dynamics are.

Background and the gap

Three bodies of work meet here. The first concerns the training horizon. Learning with Imperfect Models: When Multi-step Prediction Mitigates Compounding Error analyzes linear dynamical systems. It shows that recursive one-step models have a bias that grows with horizon when the system is only partially observed, that direct multi-step predictors avoid this bias but need many more parameters, and that a one-step model trained with a multi-step loss gets the low bias of the second at the parameter cost of the first. This paper supplies our training objective. Its analysis is linear, single-machine, and says nothing about optimization difficulty. VARNN feeds recent prediction errors back into a memory state so a regressor can adapt to drift, but it is evaluated only one step ahead. Recursive rollouts in Koochali et al. become unstable as errors compound. Both motivate the multi-step setting. Neither addresses how the horizon interacts with distributed training.

The second body concerns premature convergence. Discovering alternative solutions beyond the simplicity bias in recurrent neural networks finds that gradient descent on memory and integration tasks keeps landing on low-dimensional fixed-point attractors, meaning states the network settles into and stays at, whatever the random seed. Its deflation method (INSD) penalizes later models for resembling earlier ones and recovers higher-dimensional oscillatory solutions that generalize better to longer delays. The cost is that the models must be trained one after another. This paper both motivates our diversity measurements and serves as a baseline: we ask whether parallel horizon diversity can reach similar solutions in a single parallel run. Population methods also warn that diversity pressure is not free. Divergence-Guided Particle Swarm Optimization gains 2x to 8x on multimodal functions but does slightly worse than plain swarms on the unimodal Sphere function. We treat this as a constraint: horizon diversity must not hurt easy, nearly Markovian series. In population-based training, Doulazmi et al. show how copying the best members destroys diversity. This is part of why we keep workers on distinct objectives rather than selecting among them.

The third body concerns synchronization overhead. Exchanging gradients can take up to 50% of wall-clock time on ordinary GPU setups (Xiao et al.) and up to 92% in large distributed runs (Zhao et al.). DiLoCo scaling laws let each worker take about 30 local steps and then average parameters, using the averaged change as a "pseudo-gradient" for an outer momentum optimizer. The authors report that this acts like Lookahead and lowers C4 loss below data-parallel training at 2.4B to 10B parameters. DeMo instead sends a compressed, frequency-domain slice of each worker's momentum. WASH swaps a tiny fraction of weights between workers to keep them mergeable while preserving diversity, and was tested only on vision networks. All three keep every worker on the same objective, and any diversity among workers comes from data order or noise. None of them studies sequence models trained with rollout losses. The pairing the file flags as unexplored, rollout losses under periodic synchronization, is the starting point here. The specific new element is to make the objective itself differ across workers.

Core claim

Let $f_\theta$ be a one-step predictor with parameters $\theta$, and let $f_\theta^{(j)}$ denote applying it $j$ times to its own outputs. A worker assigned horizon $h$ minimizes the rollout loss

$$L_h(\theta) = \mathbb{E}\Big[\textstyle\sum_{j=1}^{h} \lVert y_{t+j} - f_\theta^{(j)}(y_{\le t}) \rVert^2\Big],$$

where $y_{t+j}$ is the true value $j$ steps ahead. The backward pass through an $h$-step unroll costs roughly $h$ times a one-step pass. The message exchanged at each synchronization still has $|\theta|$ entries. With $\tau$ local steps between synchronizations, bytes sent per unit of compute therefore scale as

$$\frac{\text{bytes}}{\text{FLOPs}} \propto \frac{1}{\tau\, h}.$$

This form follows directly from fixed message size and compute that grows linearly with the unroll. Long horizons and periodic averaging thus compound each other's savings.

The claim that goes beyond this accounting concerns optimization. Assign worker $k$ its own horizon $h_k$, drawn from a spread such as $\{1, 2, 4, \dots, H_{\max}\}$. After $\tau$ local steps, the workers' parameters are averaged and the outer optimizer is applied as in DiLoCo. We predict that this outperforms giving every worker the same mixed-horizon loss. The reasoning is that gradient mixing averages the landscapes at every step, whereas horizon-diverse workers each make real progress on their own landscape before being pulled back together. Short-horizon workers move quickly toward regions that fit local dynamics. Long-horizon workers resist fixed-point solutions, because those solutions fail visibly over long unrolls. Averaging pulls the shared model toward regions that are good for both. Our specific, falsifiable expectation is that horizon-diverse training yields hidden dynamics of higher dimension and better generalization beyond the training horizon than same-objective training at equal cost. We also expect it to come close to the sequential INSD models while running in parallel.

Research Program

We will build the method on a DiLoCo-style loop: an inner AdamW optimizer per worker, outer Nesterov momentum on the averaged pseudo-gradient, and 4 to 16 workers. The worker models are one-step predictors of two kinds: a gated recurrent network for the cognitive and physiological tasks, and a patch-based transformer for the long-horizon benchmarks.

The study has four parts.

The first is a controlled synthetic check on partially observed linear systems, the setting of the multi-step-prediction paper. Here the irreducible bias is known, so we can verify that the distributed horizon-diverse estimator keeps the flat-in-horizon bias that paper predicts.

The second is a set of mechanism studies on the three recurrent tasks from the simplicity-bias paper: context-dependent integration, 3-bit flip-flop, and MemoryPro. We will measure how many dimensions the hidden-state trajectories effectively use, using the participation ratio (a standard count of how many directions carry the variance). We will also locate fixed points and test accuracy when the memory delay is extended past the training range.

The third is forecasting accuracy on ETTh1, ETTm1, Weather, Electricity and Traffic at horizons from 96 to 720. We will also extend Appliances Energy, BIDMC HR and Beijing PM2.5 from VARNN's one-step setting to multi-step forecasting.

The fourth is a scale test on the numerical series of the CAF-7M corpus, whose 7.4M training samples are enough to run 16 workers without the data starvation that Decentralized Autoregressive Generation reports beyond two partitions.

We will also vary $\tau$ and the horizon spread jointly to map where workers drift too far apart to average. Weight divergence across workers is the failure mode WASH and DiLoCo both warn about. We expect wide horizon spreads to tolerate smaller $\tau$ than homogeneous training. Measuring the boundary is part of the contribution.

Baselines

The naive reference point is a one-step predictor trained with teacher forcing under standard synchronous data-parallel training with full-precision all-reduce, then rolled out recursively at test time. It shows the cost of ignoring the horizon problem altogether. A second naive point is the direct multi-head predictor with one output head per future step, trained the same way. The multi-step-prediction paper shows this model has low bias but high variance when data is limited.

The strongest accuracy competitor is the hybrid from Learning with Imperfect Models, a one-step model trained with a mixed-horizon rollout loss, run under synchronous data-parallel training. This is the best available method on the forecasting question, with communication left uncontrolled. For the systems question, the strongest competitor is the same mixed-horizon model trained under DiLoCo with the same $\tau$. On C4, DiLoCo with two replicas reaches lower evaluation loss than data-parallel training at 2.4B, 4B and 10B parameters, keeping above 80% compute utilization with 10x to 100x less bandwidth. We will also run DeMo, which on OLMo-300M sends 7.5 MB per step against 637 MB for AdamW with no loss in accuracy. And we will run a WASH-coupled variant, whose CIFAR-100 ResNet-18 averaged model reaches about 82% against 84% for the full ensemble. The file gives no reported result on Lacuna for any of these communication-efficient methods on forecasting benchmarks. For the forecasting benchmarks themselves, the file reports only relative gains: the dual-domain MoE transformer cuts MSE by 50.9% to 56.9% relative to PatchTST and TimesNet, and VARNN improves on classical baselines by about 35%, 44% and 47% on its three datasets. No absolute MSE values appear in the file, so we will rerun PatchTST and the MoE transformer under our protocol rather than quote numbers.

For the diversity question, the strongest competitor is INSD, run sequentially with its three models. We will report both its accuracy and its serial wall-clock time.

The ablation that isolates our key ingredient holds everything fixed except how horizons are spread. It compares three arms with identical per-worker FLOPs, data, $\tau$, outer optimizer, and bytes sent. In the first, every worker uses the same mixed-horizon loss. In the second, horizons are fixed per worker (our method). In the third, horizons are reassigned at random each round, which separates horizon diversity from persistent worker identity. All methods get the same hyperparameter search budget, about 30 trials over learning rates and outer momentum, selected on validation data. Beating the strongest competitor means lower test MSE averaged over horizons 96 to 720, and higher extended-delay accuracy on MemoryPro, than mixed-horizon DiLoCo, with non-overlapping 95% intervals over five seeds. It also means matching synchronous mixed-horizon training to within 2% MSE while sending at least 10x fewer bytes.

Why not just…?

Why not just train every worker on a mixed-horizon loss and skip the heterogeneity? This is the real threat. The bytes-per-compute argument above applies equally to it, so the communication savings alone do not favor our method. The only difference is whether diversity lives inside each gradient or across workers between synchronizations. The equal-cost three-arm ablation settles this directly. If mixed-horizon DiLoCo matches horizon-diverse DiLoCo on long-horizon MSE and on hidden-state dimensionality, we will report that horizon diversity adds nothing. The remaining contribution would then be the accounting result and the drift map.

Why not just use a horizon curriculum, training short then long? Curricula fix the order in time but still move a single trajectory through one landscape at a time, so they share the same exposure to fixed-point collapse. The simplicity-bias paper finds that recurrent networks fall into fixed points regardless of seed. Whether a curriculum escapes them is untested in the file. We include a curriculum arm at equal compute and compare participation ratio and extended-delay accuracy.

Why not just rely on independent seeds, or on WASH-style weight swapping, for diversity? The simplicity-bias paper reports that seeds alone keep producing the same low-dimensional solution. WASH-style swapping keeps workers in one basin by design, and WASH has never been tested on recurrent dynamics. Diversity driven by noise or by data order has no reason to push toward richer dynamics, whereas long-horizon objectives penalize fixed points directly. The WASH-coupled arm will show whether weight swapping disrupts recurrent memory and whether it adds anything on top of horizon diversity.

Why not just compress gradients with DeMo and keep ordinary one-step training? Compression reduces bytes but does not change the objective. The recursive bias in partially observed systems therefore remains, and the multi-step-prediction paper shows it grows with horizon. DeMo is also orthogonal to our method: we run DeMo-compressed horizon-diverse training as an additional arm, to test whether the savings stack.

Decisive checks

Three outcomes would settle the proposal.

The first is the equal-cost comparison against mixed-horizon DiLoCo. If horizon-diverse workers do not win on long-horizon MSE across the five forecasting datasets, or on extended-delay MemoryPro accuracy, the central claim fails.

The second is the mechanism. Horizon-diverse training should produce hidden dynamics with a participation ratio and fixed-point structure closer to INSD's alternative models than to standard training, and it should do so in one parallel run. If the dimensionality matches standard training, the proposed explanation is wrong even if accuracy improves.

The third is the unimodal guard. On nearly Markovian series, and on the synthetic system with full observability, horizon diversity should cost no more than a few percent against single-step training. The swarm literature's Sphere result warns that it might. If the cost is larger, we will report the regime boundary, so practitioners know when to keep every worker on the short horizon.

Throughout, we will report bytes sent and wall-clock time under simulated bandwidth limits alongside accuracy, so the communication claim can be checked as directly as the optimization claim.