A segmentation model trained on video-game driving scenes and deployed on real roads will meet real conditions one after another: a European city, then a crowded Indian street, then fog, then night. Methods that adapt a model during deployment without labels stay stable over such streams by pulling the model back toward its original training state. CoTTA restores a fraction of weights to the source checkpoint, and DPCore aligns feature statistics to source references. When the source is a simulator, that original state is exactly what we want to escape, because it encodes rendering quirks and other simulator shortcuts. We propose to test a simple consequence of this. A synthetic anchor should make sim-to-real adaptation both worse on average and more sensitive to the order in which real domains arrive, since whichever real domain comes first decides which shortcuts get unlearned before the anchor drags the model back.
The fix we study is to retire the synthetic anchor as soon as real data exists. The replacement anchor is built only from real domains seen so far, and it is computed by a symmetric average, so it does not depend on arrival order. In two sentences: stabilizing a sim-trained model during deployment by tying it to its synthetic origin re-injects simulator shortcuts and makes outcomes depend on domain order. Tying it instead to an order-free summary of the real domains already visited should remove both problems at almost no cost.
Core Claim
Our hypothesis is that the order sensitivity of continual test-time adaptation is measurably larger when the source is synthetic than when it is real, and that this gap comes from the anchor rather than from the adaptation rule. We test it in the setting where a model adapts online to unlabeled test data, keeps no source images, and receives no domain labels. The concrete mechanism keeps per-domain feature statistics (layer-wise means and standard deviations, with no images stored). The anchor is defined as their barycenter, meaning the average distribution under the 2-Wasserstein distance. For diagonal Gaussians this reduces to averaging means and averaging standard deviations:
$$\bar{\mu} = \frac{1}{K}\sum_{k=1}^{K}\mu_k, \qquad \bar{s} = \frac{1}{K}\sum_{k=1}^{K}s_k,$$
Here $K$ is the number of real domains identified so far, and $\mu_k, s_k$ are the stored feature mean and standard deviation for domain $k$. Both are symmetric functions of the stored domains, so the anchor itself is permutation-invariant. The synthetic source statistics are used only until the first real domain has been summarized. Per-domain prompts are then fitted against $(\bar{\mu}, \bar{s})$ rather than against synthetic references.
The anchor alone does not make the whole system order-invariant, because prompts fitted early saw an earlier anchor. Our narrower claim is that prompts for revisited domains should converge to the same fixed point regardless of order. This predicts that order disparity shrinks after each full cycle through the domains, which is directly testable.
Relation to Prior Work
Three works motivate the problem. Continual Test-Time Domain Adaptation (CoTTA) introduced stochastic restoration of 1% of weights to the source, but tested only repeated loops of one weather sequence, never permutations. DPCore freezes a vision transformer, which is a network built from attention layers. It learns small input prompts stored in a coreset, a bank of prompts indexed by domain statistics, and it argues theoretically that this design is order-independent. However, it aligns everything to source statistics and was evaluated on corruptions and real-to-real weather, not synthetic-to-real. Continual Unsupervised Domain Adaptation with Adversarial Learning (ETM) is the closest sim-to-real continual work, running GTA5 → Cityscapes → IDD with one memory module per target. It needs the synthetic source data throughout and reports only one order.
Our work is DPCore-like in machinery. It differs in what the stable reference point is, and in measuring order sensitivity in a regime where that reference point is contaminated.
Two works supply method. We borrow the order-sensitivity metric of Scalable and Order-robust Continual Learning with Additive Parameter Decomposition, the Average Order-normalized Performance Disparity. APD reports 1.79% on CIFAR-100 against 7–8% for baselines, but in a task-labeled classification setting with no domain gap. We borrow the ordering principles of Optimal Task Order for Continual Learning of Multiple Tasks, which derives "maximally dissimilar consecutive tasks" and "outliers first, typical tasks last" rules in linear models. That work finds the best order up to seven times better than the worst. We do not assume its linear theory holds here. We use its zero-shot similarity estimates to construct adversarial and favorable orders as stress tests, which also checks whether its rules survive in deep sim-to-real adaptation.
One work constrains the design. Tent shows that entropy minimization on normalization parameters helps stationary GTA5 → Cityscapes transfer, raising IoU from 28.8% to 35.8%, but CoTTA shows it accumulates errors on continual streams. Any proposal must therefore beat it on streams, not on single steps.
Research Program
The first stage is measurement. We train one transformer segmentation model on GTA5 and a matched model on clear-weather Cityscapes. Each is deployed on a stream of real domains: Cityscapes and IDD for the synthetic-source model, and the four ACDC conditions (fog, night, rain, snow) for both. We run CoTTA, DPCore, and Tent-continual over many permutations of the domain order and compute final per-domain mIoU and order disparity.
We also add a shortcut probe. A linear classifier is trained to separate GTA5 features from real features, and we track how separable the adapted model's features remain from synthetic ones over the stream. If synthetic anchoring re-injects shortcuts, the probe should show features drifting back toward the synthetic cluster after each anchor alignment.
The second stage builds the real-anchored variant. It needs one mechanism for deciding when a new real domain has begun without labels. We reuse DPCore's statistical-distance threshold, and we add a per-sample assignment step that sends each sample in a batch to its nearest stored domain. This targets DPCore's documented failure on mixed-domain batches. We also build a CoTTA counterpart in which the 1% weight restoration targets a symmetric running average of per-domain adapted weights instead of the synthetic checkpoint. This tests whether the anchor effect holds across adaptation rules.
The third stage analyzes why order matters when it does. We use Li and Hiratani's zero-shot transfer matrix, built from roughly 1% of each domain, to predict which orders are worst. We then check whether real anchoring flattens the gap between the predicted worst and best orders.
Baselines
The do-nothing reference is the GTA5-trained model applied frozen to every domain. It is trivially order-invariant and fixes the floor. VisDA reports a source-only segmentation baseline of 21.4 mIoU on its GTA5-to-real track, though with an older backbone.
Episodic Tent, which resets the model after every batch, is the cheapest order-invariant adaptive baseline. Tent-continual is the cheapest order-dependent one.
The strongest competitor is DPCore, run from its public code (yunbeizhang/DPCore). It already targets order robustness, uses a frozen backbone, and is the method our variant modifies. On ImageNet-C under dynamic recurring shifts it reports 42.1% error against 52.1% for ViDA. We found no reported result on Lacuna for DPCore on Cityscapes-to-ACDC or any synthetic-to-real stream, so we produce those numbers ourselves.
CoTTA is the second strong competitor. It reports 58.6 mIoU on Cityscapes-to-ACDC over 10 cycles, and Hybrid-TTA reports 62.2% mIoU on the same benchmark. Both numbers serve as sanity checks for our real-source runs. ETM is a non-source-free reference on GTA5 → Cityscapes → IDD. It reports a 1.35-point Cityscapes drop after IDD, against 7.41 for AdaptSegNet. We rerun it in both orders.
Fairness rules apply to every method. All methods share the same backbone, batch size, stream, and number of gradient steps per batch. Each method gets the same hyperparameter budget, tuned on one held-out order and frozen across all others.
The key ablations swap only the anchor while holding the rest of the system fixed. The four anchors are synthetic source statistics (which is DPCore itself), statistics of whichever real domain arrived first, our barycenter, and an oracle anchor computed from all real domains pooled. The oracle is the ceiling for any real anchor.
Beating DPCore means two things on the synthetic-source stream: lower order disparity across permutations, and equal or higher mean mIoU. We also require that the gain over DPCore be clearly smaller on the real-source stream. That difference is what attributes the effect to shortcut contamination rather than generic regularization.
Why not just…?
Why not just use DPCore, which already claims order independence? This is the main threat. If DPCore's order disparity on the synthetic-source stream is already near its real-source level, our hypothesis is wrong. The first-stage measurement is designed to show exactly that, and we would report it as a negative result rather than proceed to the method. DPCore's guarantee concerns retrieval of stored prompts. It does not address what the prompts are aligned to, so the claim can hold while the alignment target still varies in quality.
Why not just adapt offline to the first real domain and use that as the new source? This works whenever an unlabeled real dataset exists upfront, and it is our "first-real anchor" ablation. Its weakness is built in: the anchor is whichever real domain happened to arrive first. Permuting the first domain measures that dependence directly. If first-real anchoring matches the barycenter across permutations, the symmetric construction adds nothing, and we would say so.
Why not just train a less shortcut-prone source model with heavy randomization or stylization? LoopSR finds wide randomization costly in robotics: zero-shot RMA took 6.97s on stairs against 4.33s with targeted adaptation. VisDA shows synthetic-only models falling from 99% to 28% accuracy on real images. Better source training is complementary, not competing. We will repeat the comparison with a style-randomized source. The prediction is that the benefit of real anchoring shrinks as source contamination drops, which would strengthen the causal story.
Why not just replay a buffer of real images? Lifelong Language Learning with Adapter based Transformers reports that replay stays susceptible to sequencing bias. Storing raw road imagery also conflicts with the source-free, low-memory constraints of deployment. Our method stores only per-layer statistics.
Proposed Experiments
The primary benchmark is the GTA5-trained model on a seven-domain real stream: Cityscapes, IDD, and the four ACDC conditions. We sample 24 orders uniformly. We add the worst and best orders predicted by Li and Hiratani's rules, plus recurring versions that cycle through all domains three times. The control is the Cityscapes-trained model on ACDC permutations. A secondary classification check uses VisDA-C, splitting real target images into the COCO-derived validation and YouTube-derived test domains as a two-domain stream in both orders.
The metrics are mean final mIoU, per-domain forgetting, order disparity in the APD style, the drift of the shortcut probe, and throughput.
Three results are decisive. The first is whether order disparity under synthetic anchoring exceeds the real-source control by a margin larger than seed variance. If not, the premise fails. The second is whether the barycenter anchor closes most of the distance between the synthetic anchor and the pooled-real oracle, in both disparity and mean mIoU. The third is whether disparity among revisited domains falls cycle over cycle under real anchoring but not under synthetic anchoring, which is the fixed-point prediction. A further mixed-batch test, which blends Cityscapes and IDD frames within batches, checks that per-sample assignment prevents the spurious prompt creation documented for DPCore.
References
- Continual Test-Time Domain Adaptation
- DPCore: Dynamic Prompt Coreset for Continual Test-Time Adaptation
- Continual Unsupervised Domain Adaptation with Adversarial Learning
- Scalable and Order-robust Continual Learning with Additive Parameter Decomposition
- Optimal Task Order for Continual Learning of Multiple Tasks
- Tent: Fully Test-Time Adaptation by Entropy Minimization
- VisDA: The Visual Domain Adaptation Challenge
- Hybrid-TTA: Continual Test-time Adaptation via Dynamic Domain Shift Detection
- LoopSR: Looping Sim-and-Real for Lifelong Policy Adaptation of Legged Robots
- Lifelong Language Learning with Adapter based Transformers