Abstract
Unsupervised Domain Adaptation (UDA) for video semantic segmentation aims to leverage synthetic data to train models for real-world applications without requiring manual labels for target domains. However, traditional UDA methods often suffer from class bias—favoring frequent classes like roads over infrequent "hard" classes like fences—and temporal inconsistency when applying image-level augmentations to video. This paper introduces a novel framework featuring Cross-Domain Moving Object Mixing (CMOM) and Feature Alignment with Temporal Context (FATC). CMOM preserves temporal logic by mixing moving objects and their corresponding optical flows across domains, while FATC utilizes temporal consistency to filter noisy pseudo-labels. The proposed approach achieves state-of-the-art results on major benchmarks, significantly improving the recognition of thin structures and small objects.
Introduction
Training deep neural networks for video semantic segmentation is notoriously data-intensive, requiring pixel-level annotations for every frame. Synthetic datasets, such as those generated from high-fidelity video games, offer a cost-effective alternative. Despite their utility, a significant "domain gap" exists between synthetic textures and real-world environments, causing models to perform poorly on actual footage.
Current UDA techniques typically rely on self-training, where a model generates "pseudo-labels" for real-world data. This process often leads to a bias where the model excels at identifying common classes (e.g., sky, road) but fails to recognize infrequent, complex classes (e.g., traffic signs, poles, fences). Furthermore, standard data augmentation techniques like "CutMix" often break temporal continuity in videos, as objects may appear to teleport or vanish between frames, confusing the model's understanding of motion.
Methodology
Cross-Domain Moving Object Mixing (CMOM)
To address the loss of temporal logic, the authors propose CMOM. Unlike static image mixing, CMOM identifies moving objects in the source (synthetic) domain and pastes them into the target (real) domain across consecutive frames. Crucially, the same mask is applied to both the RGB frames and the underlying optical flow. This ensures that the synthetic object's motion remains physically plausible within the real-world scene, providing the model with consistent temporal cues for "hard-to-transfer" classes.
Feature Alignment with Temporal Context (FATC)
Feature alignment is often used to bridge the domain gap, but it is highly sensitive to noisy pseudo-labels. FATC serves as a denoising mechanism by leveraging temporal consistency. The framework compares the model's current prediction with a version of the previous frame's prediction warped by optical flow. If the predictions are inconsistent across time, the area is deemed unreliable and excluded from the alignment process. This ensures the model only learns from high-confidence, temporally stable features.
Experimental Results
The framework was evaluated on standard synthetic-to-real benchmarks, specifically transitioning from VIPER (GTA V) and SYNTHIA-Seq to Cityscapes-Seq.
- Performance Gains: On the VIPER $\rightarrow$ Cityscapes-Seq task, the method achieved 53.81% mIoU, a substantial improvement over the previous state-of-the-art (48.9%). Similarly, it reached 56.31% mIoU on the SYNTHIA-Seq benchmark.
- Qualitative Stability: Visual analysis confirms that the model is significantly more stable than its predecessors. It successfully avoids common pitfalls, such as misclassifying sidewalks as terrain or failing to detect thin structures like fences and poles.
- Efficiency: Despite its effectiveness, CMOM is computationally efficient, adding only a marginal 2.8% overhead to the total training time.
Conclusion
By integrating temporal physics into data augmentation and using time as a natural supervisor for denoising, this framework sets a new standard for video UDA. While the method currently relies on pre-trained optical flow models—which may struggle in extreme conditions—it demonstrates a powerful path forward for training robust video models without the need for exhaustive human labeling. Future work may explore the application of these techniques to more diverse video domains beyond urban driving scenes.
Summary updated