A street-scene segmentation model, meaning one that labels every pixel as road, car, pedestrian and so on, is cheap to train on video-game renders such as GTA5, where every pixel comes with a free and exact label. Once it is deployed on a real car, though, it sees a stream of real conditions it was never trained on: clear city streets, then fog, night, rain and snow, with no labels. Today's methods for adapting a model to such a stream without labels, called continual test-time adaptation, keep the model from drifting by pulling its weights back toward the model it started from. When that starting model was trained in a simulator, this pull drags it back toward the simulator's look: its textures, lighting and rendering quirks.
We propose a different anchor. The simulator is wrong about appearance but right about semantics and scene layout, and its labels never go stale. During deployment we keep replaying labeled synthetic images, but before they reach the model's early layers we re-dress them in the appearance statistics of whatever real condition the car is currently in. The anchor then teaches "what a car is" in tonight's lighting rather than in GTA5's lighting. The question this work answers is whether the simulator, used this way as a privacy-free labeled memory, beats both weight restoration and plain synthetic replay at preventing forgetting and error buildup across a long real-world stream.
Core Claim
In continual adaptation that starts from a synthetic-trained model, the thing that should stay fixed is the simulator's labels, not the simulator's weights or the simulator's appearance. A supervised loss on synthetic images whose low-level feature statistics have been swapped to match the current real batch gives a stabilizing signal that agrees with the real-stream adaptation signal rather than fighting it. Plain synthetic replay supplies correct labels but pulls early features back toward synthetic appearance. Weight restoration pulls everything back. Re-dressed replay should keep semantic competence across the stream without re-injecting simulator bias.
This claim is testable as a mechanism, not only as a final score. We predict that the cosine similarity between the gradient of the synthetic supervised loss and the gradient of the real self-training loss is negative or near zero for plain synthetic replay, and becomes positive after re-dressing. If it does not, the method has no reason to work, and the experiments will show that.
Research Program
The setting is built from benchmarks in the file. A segmentation network is trained only on GTA5 (Playing for Data), with SYNTHIA (The SYNTHIA Dataset) as a second source for replication. It is then deployed without labels on an online real stream. The stream begins with Cityscapes (The Cityscapes Dataset) and continues through the ACDC Fog, Night, Rain and Snow domains (ACDC), cycled ten times as in the CoTTA protocol.
The adaptation core is held fixed: a mean-teacher self-training loop of the CoTTA kind (Continual Test-Time Domain Adaptation), in which a slowly averaged copy of the model produces pseudo-labels for the current model. That way only the anchor differs between conditions. At each step the model also sees a small batch drawn from a fixed bank of labeled synthetic images, a few thousand GTA5 frames. The bank never changes, so unlike learned generative replay it cannot itself forget or drift.
Re-dressing works in feature space and costs almost nothing. We keep a running average of per-channel mean and standard deviation of early-encoder features on the real stream. Synthetic replay features at those layers are renormalized to those statistics before the rest of the forward pass. This is the adaptive instance normalization operation that GarDA (Generative appearance replay for continual unsupervised domain adaptation) uses inside a trained GAN, here applied directly without any generator. Which layers to re-dress is the main design variable. Shallow layers carry appearance; deep layers carry semantics and should not be touched. We sweep the cut point.
GarDA supplies the method family, but it replays past real domains through a GAN costing hours per domain. Our replay content is simulator renders and our styling is a statistic swap. The analysis side tracks the gradient-agreement statistic above per layer and per domain. It also tracks entropy of predictions on replayed versus real data, because Dealing with Synthetic Data Contamination in Online Continual Learning shows that synthetic images in a stream produce overconfident, tightly clustered features. That is the failure we must check we are not reproducing.
Baselines
The naive reference is the GTA5-trained model frozen and evaluated on the stream with no adaptation, alongside online batch-normalization statistic updates and TENT-style entropy minimization. In the real-source setting CoTTA reports these erode under long streams: on CIFAR10-C, batch-norm adaptation reaches 20.4% error and TENT-continual 20.7%, against 16.2% for CoTTA (Continual Test-Time Domain Adaptation). The open question is how much worse they do from a synthetic source. All baselines share the backbone, the stream order, the number of updates per incoming batch and the hyperparameter search budget. Hyperparameters are selected on a held-out ordering using SYNTHIA as source, never on the GTA5 test stream.
The strongest existing method is CoTTA itself, with stochastic restoration of 1% of weights to the source checkpoint. With a real Cityscapes source on the ten-cycle Cityscapes-to-ACDC stream it reports 58.6 mIoU (Continual Test-Time Domain Adaptation). That is a different setting. We found no reported result on Lacuna for CoTTA or any continual test-time method starting from GTA5 or SYNTHIA, so we will run it ourselves. We will also run KFF's prompt-pool approach (Class-aware Domain Knowledge Fusion and Fission for Continual Test-Time Adaptation), which reports 34.8% ImageNet-C error against 39.9% for DPCore but was only tested on synthetic corruptions. Beating these means higher mean online mIoU averaged over all ten cycles and lower forgetting at matched compute. We count compute in forward passes, which favors us, since CoTTA uses up to 32 augmentations per step.
The ablation that isolates the key ingredient is plain synthetic replay: the same labeled bank and loss weight, with re-dressing switched off. Two further ablations bracket it. One applies re-dressing at all layers, which should destroy semantics. The other combines re-dressed replay with weight restoration, which asks whether the two anchors are redundant. As an upper reference we cite offline unsupervised adaptation with full access to unlabeled Cityscapes, which reaches 60.2 mIoU from GTA5 (Unsupervised Contrastive Domain Adaptation for Semantic Segmentation). We also cite fully supervised HRNet dropping from about 85 mIoU on Cityscapes to 75.0 on ACDC (ACDC).
Why not just…?
Why not just replay the raw synthetic images with their labels? This is the real threat. If plain replay matches re-dressed replay, the contribution shrinks to "carry your simulator data with you," which is still a useful finding but not this paper. The evidence that it may fall short is indirect. Synthetic images in online streams yield low-entropy, separated features that hurt real accuracy (Dealing with Synthetic Data Contamination in Online Continual Learning). Fine-tuning on a narrow stream while anchored to simulation helped in The SwaNNFlight System, but only in low-dimensional control where there is no appearance gap. The direct comparison and the gradient-agreement measurement settle it. If re-dressing does not raise gradient agreement and mIoU on Night and Snow, we report that plainly.
Why not just keep CoTTA's weight restoration? Restoring weights to a GTA5 checkpoint restores GTA5's appearance features along with its semantics. Nothing in CoTTA's evidence covers this case, because its source was real Cityscapes. We test it head on by running CoTTA from both a Cityscapes source and a GTA5 source on the same stream. The gap between the two quantifies the cost of restoring to a synthetic checkpoint. We then check whether re-dressed replay closes it.
Why not just randomize appearance heavily offline and deploy a frozen model? Domain randomization trades nominal performance for breadth, as LoopSR reports for locomotion (90% versus 60% pit success against the adaptation-based RMA). ACDC's authors note that methods tuned on synthetic weather often fail on real adverse weather. We include a frozen model trained on GTA5 with strong photometric and style randomization as a baseline. If it matches our online method, adaptation is unnecessary for this stream, and that would be worth knowing.
Why not just use a GAN to restyle, as GarDA does? GarDA's generator costs hours per domain and can hallucinate in safety-relevant regions. A feature-statistic swap has neither cost. We compare against offline-restyled synthetic replay under a matched wall-clock budget to see whether the GAN buys anything.
Proposed Experiments
The primary result is mean online mIoU over ten cycles of Cityscapes then ACDC Fog, Night, Rain and Snow, from GTA5 and separately from SYNTHIA. It is paired with two forgetting measures. The first is mIoU on held-out Cityscapes validation after each cycle. The second is the drop in each ACDC domain's mIoU between its first and last visit.
Three checks decide the claim. The first is whether re-dressed replay beats plain replay on the domains farthest from the simulator's look, Night and Snow, by more than seed variance over three orderings. The second is whether CoTTA from a synthetic source loses more relative to its real-source version than our method does. The third is whether gradient agreement between replay and stream losses at the shallow layers turns positive with re-dressing and tracks the per-domain gains. A clean negative on the third check, with mIoU gains anyway, would mean the method works for reasons other than the one we propose, and we would say so.
A secondary experiment varies the replay bank size from a few hundred to several thousand frames. It also replaces the fixed bank with fresh renders from SYNTHIA's season and weather variants. This tells us whether the simulator's ability to generate new labeled content adds anything beyond a fixed stored set, which matters for the broader idea of a simulator as a growing replay memory.
References
- Playing for Data: Ground Truth from Computer Games
- The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes
- The Cityscapes Dataset for Semantic Urban Scene Understanding
- ACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding
- Continual Test-Time Domain Adaptation
- Generative appearance replay for continual unsupervised domain adaptation
- Dealing with Synthetic Data Contamination in Online Continual Learning
- Class-aware Domain Knowledge Fusion and Fission for Continual Test-Time Adaptation
- Unsupervised Contrastive Domain Adaptation for Semantic Segmentation
- The SwaNNFlight System: On-the-Fly Sim-to-Real Adaptation via Anchored Learning
- LoopSR: Looping Sim-and-Real for Lifelong Policy Adaptation of Legged Robots