Forecasting models for traffic, crops, weather or public health are usually trained to minimize average error over every location at once. They look good on aggregate benchmarks and then fail badly on the locations that behave differently: volatile intersections, drought-prone counties, rural areas with sparse data. The methods that fix this all run an outer loop around training. They fit a model, measure where it fails, regroup the locations into specialized partitions, and fit again. Each pass retrains from scratch or re-evaluates every location, so the cost of fixing spatial unevenness grows with the number of passes times the size of the network.
We propose to make that outer loop incremental. One shared forecasting network is trained once. Each round of the loop then fits only small low-rank add-on weights for each group of locations, starting from the previous round's weights. It revisits only the locations whose group membership is actually in doubt, and it stops when the ranking of locations by error stops changing. The claim to test is that most of the hard-region benefit of iterative specialization comes from a few early, cheap rounds. If so, an incremental loop should match full iterative methods on the worst-served locations at a small fraction of their compute. The note also takes seriously the possibility that no loop is needed at all.
Core Claim
Several iterative methods target spatial heterogeneity, meaning the fact that the process generating the data differs from place to place. Each of them treats every round as a fresh optimization problem over all locations.
Task-Adaptive Meta-Learning Framework for Advancing Spatial Generalizability (Adaptive-MAML-B) handles county-level crop yield. It trains a meta-learned model, splits counties into easy and hard groups by minimizing error variance within each group, and recursively trains new meta-parameters down a tree. Every split nests a second-order meta-learning loop.
A Spatio-temporal Cluster-aware Supervised Learning Framework for Predicting County-level Drug Overdose Deaths (CASL) alternates between cluster assignment and refitting per-cluster regressions over all counties at every step. It is restricted to linear autoregressive models.
TESTAM routes each road segment to one of three expert networks. It trains the routing component against pseudo-labels, which requires running every expert on every node to rank them.
These three papers motivate the proposal: they show that specialization helps hard regions. Their per-round cost is also exactly the overhead we target.
Our hypothesis is that, after the first round, group assignments are stable for most locations. The information a loop adds is concentrated at the boundary between groups. A loop that warm-starts small per-group weights and touches only boundary locations should therefore keep the hard-region gains. In the idealized case, a location enters a later round only if its error rank moved across a group boundary. The loop's total cost then behaves like
$$ C_{\text{inc}} \approx C_{\text{base}} + \sum_{r=1}^{R} \left( c_{\text{ad}}\,|B_r| + c_{\text{fit}} \right), $$
where $C_{\text{base}}$ is one training run of the shared network, $R$ is the number of rounds, $|B_r|$ is the number of boundary locations re-scored in round $r$, $c_{\text{ad}}$ is the per-location cost of evaluating the add-on weights, and $c_{\text{fit}}$ is the cost of refitting the per-group add-on weights. A full-restart loop instead costs roughly $R \cdot C_{\text{base}}$, or more for nested meta-learning. The saving comes from $|B_r|$ shrinking relative to $N$, the total number of locations. Measuring whether it actually shrinks is part of the study.
Research Program
Shared network. We train one shared forecasting network per domain with ordinary uniform-risk training. For traffic this is Graph WaveNet, a graph convolutional model with a learned adjacency. For crop yield it is an LSTM matched to the backbone used by Adaptive-MAML-B.
Per-group add-on weights. We then freeze the shared network. Each group of locations receives low-rank add-on weights (LoRA-style adapters, small trainable matrix corrections to the frozen layers). This reuses the mechanism that LM-WEATHER applies per weather station in a federated setting, where clients share only 0.38M parameters. We use the same mechanism per spatial group rather than per device.
Initial grouping. Groups are initialized once, by splitting locations on first-round error in the variance-minimizing way Adaptive-MAML-B uses.
Incremental rounds. Each later round does four things. It warm-starts every group's adapter from the previous round. It identifies boundary locations, whose error rank lies within a margin of a group threshold. It re-scores only those locations under neighbouring group adapters. It reassigns a location if another adapter fits it better. Splitting a group, which grows the tree, is allowed only when that group's within-group error variance exceeds a threshold. The split is then fitted from the parent adapter, not from scratch.
Stopping rule. We borrow the ranking-stability criterion from An Iterative Method for Unsupervised Robust Anomaly Detection Under Data Contamination. That paper stops an outer reweighting loop once sample scores stop drifting across a percentile boundary. Here the loop stops once the share of locations changing group falls below a small tolerance, applied to location error ranks rather than anomaly scores. This makes the number of rounds data-driven rather than a hyperparameter.
Risk from warm-starting. Warm-starting carries a known risk. Training Deep Neural Networks in Generations resets parameters between generations precisely to avoid accumulating bias, and self-training work reports error cascades across rounds. We address this directly. The shared network is never updated inside the loop, so only small adapters can drift. We also include a cold-restart-adapter ablation to measure any cascade.
New locations. A side benefit is that a newly installed sensor can be placed by scoring it under the existing group adapters. This needs no retraining, which the strictly transductive node embeddings of G-SWaN require, and no generative replay of the kind TFMoE uses. We test this on PEMSD3-Stream as a secondary result, not the core claim.
Baselines
Naive reference. The shared network trained once with uniform risk, reported on overall and worst-location error. On crop yield, the file's closest analogue is the traditional LSTM, which reaches R² of 71.34% overall and 37.96% on hard low-yield counties (Task-Adaptive Meta-Learning). This establishes how much unevenness exists to fix.
Strongest iterative competitor, crop yield. Adaptive-MAML-B reports R² of 80.58% overall against 78.98% for standard meta-learning, and 64.39% on hard counties (Task-Adaptive Meta-Learning). We will rerun it on the same USDA-NASS/Ecosys splits with the same backbone and hyperparameter search budget, and log its GPU-hours. Beating it means matching hard-county R² within noise, measured over five seeds, at clearly lower total training compute. Exceeding its accuracy is not required, since the claim is about cost.
Strongest competitor on traffic unevenness. FairSTG reduces the variance of per-location mean absolute error by 5.59% to 8.15% on METR-LA and PEMS-BAY. It does this through mix-up with retrieved "easy" samples and a variance penalty, and it accepts a small loss in average accuracy. We run it on the same Graph WaveNet backbone with an equal tuning budget. For the average-accuracy reference point we use Graph WaveNet's reported 60-minute METR-LA error of 3.53, against 3.60 for DCRNN. Beating FairSTG means an equal or larger variance reduction with no loss in average error.
Routing competitor. TESTAM runs on METR-LA and EXPY-TKY, where it is reported to outperform 13 baselines including MegaCRN. MegaCRN's reported 60-minute EXPY-TKY figures are an error of 6.83, root-mean-square error of 11.04 and percentage error of 31.02%. No reported per-horizon TESTAM number appears in the file, so we will reproduce it ourselves.
Transductive embedding competitor. A node-embedding-conditioned version of the shared network trained in one pass, in the style of G-SWaN or of HN-MVTS, which generates channel-specific final weights at only 5–25% extra training time per epoch.
Ablations. These isolate the key ingredient. The first is full-restart rounds with every location re-scored, which is our method with incrementality removed. The second is cold-start adapters each round. The third is a fixed round count instead of the stability rule.
Why not just…?
Why not just give every location a learned embedding and train once? This is the most serious threat, because it has no loop at all. G-SWaN's embeddings encode enough location identity that a linear probe recovers latitude and longitude with R² of 0.996. HN-MVTS shows per-channel weight generation is cheap. What the file does not show is that embeddings rescue the hardest locations. Uniform-risk training with embeddings still weights easy locations by their numbers, and FairSTG's results show that aggregate-optimized traffic models remain uneven. The direct comparison on worst-30% error settles it. If the embedding model matches our loop on hard locations, we will report that the iterative specialization literature is paying for nothing on these benchmarks. That is still a useful result.
Why not just reweight hard samples in a single training run? FairSTG does roughly this and gains variance reduction. It also documents a loss in average accuracy, and it adds a nearest-neighbour retrieval step inside training. Our design keeps the shared network trained on uniform risk and specializes only the adapters, so average accuracy should not suffer. The FairSTG comparison tests exactly this trade-off.
Why not just cluster once on location features and stop? One-shot clustering costs nothing extra, and TFMoE partly does this with an offline clustering stage. Its complexity, however, is quadratic in the number of sensors. Its clusters are also fixed before forecasting error is known, whereas the adaptive methods split on observed error. Our round-one-only ablation is exactly "partition once and freeze." If it matches the stopping-rule version, the incremental loop reduces to one cheap round, and we will say so.
Why not just run the full loop for two rounds? This is a real alternative to our stopping rule. We compare fixed-round budgets against the stability rule at equal compute. The rule earns its place only if it avoids both under- and over-iteration across all four domains without per-dataset tuning.
Proposed Experiments
Datasets and outcome. We use USDA-NASS/Ecosys crop yield (630 counties), METR-LA, PEMS-BAY, EXPY-TKY (1,843 road links, the scale test) and the CDC county overdose benchmark. On the CDC data, CASL's reported error of 14.60 on unsuppressed counties serves as the linear alternating-minimization reference, and we adopt its hinge loss for suppressed counts. Every method is reported on four measures: overall error, error on the worst 30% of locations, per-location error variance, and total training GPU-hours including all outer-loop rounds.
Diagnostic 1: boundary shrinkage. We plot the boundary set size $|B_r|/N$ per round. The cost argument holds only if this falls quickly, say below 10% by round two. If assignments churn, the method loses its reason to exist.
Diagnostic 2: the compute–accuracy frontier. This is decisive. We place our incremental loop, the full-restart ablation, Adaptive-MAML-B and the embedding baseline on a plot of hard-location accuracy against GPU-hours. The proposal succeeds if the incremental loop sits on the frontier: it must match full iterative specialization on hard locations at markedly lower cost, and beat the single-pass embedding model on hard locations at comparable cost.
Diagnostic 3: error cascade. We compare warm-start and cold-start adapters on hard-location error across rounds. This checks whether reusing work across rounds carries the bias that the generational-training paper warned about.
Secondary test: new sensors. On PEMSD3-Stream, we assign newly installed sensors by scoring them under the existing adapters. We compare against TFMoE's reported 60-minute error of 17.20 and its roughly 244-second update time, against more than 940 seconds for full retraining.
References
- Task-Adaptive Meta-Learning Framework for Advancing Spatial Generalizability
- A Spatio-temporal Cluster-aware Supervised Learning Framework for Predicting County-level Drug Overdose Deaths
- TESTAM: A Time-Enhanced Spatio-Temporal Attention Model with Mixture of Experts
- Graph WaveNet for Deep Spatial-Temporal Graph Modeling
- Personalized Adapter for Large Meteorology Model on Devices: Towards Weather Foundation Models
- An Iterative Method for Unsupervised Robust Anomaly Detection Under Data Contamination
- Training Deep Neural Networks in Generations: A More Tolerant Teacher Educates Better Students
- Because Every Sensor Is Unique, so Is Every Pair: Handling Dynamicity in Traffic Forecasting
- Continual Traffic Forecasting via Mixture of Experts
- FairSTG: Countering Performance Heterogeneity via Collaborative Sample-Level Optimization
- HN-MVTS: HyperNetwork-based Multivariate Time Series Forecasting