Abstract
Semantic segmentation models trained on synthetic data often fail to generalize to real-world environments due to the "Sim-2-Real" domain gap. We present Multi-Resolution Feature Perturbation (MRFP), a plug-and-play training strategy that addresses this gap by targeting both low-frequency style information and high-frequency texture details. By selectively perturbing features across the frequency spectrum, MRFP forces models to learn domain-agnostic representations. Our method achieves significant performance gains on diverse real-world datasets with zero additional inference cost.
Introduction
Deep learning for semantic segmentation requires vast amounts of pixel-level annotations, which are expensive and time-consuming to produce manually. Synthetic data from high-fidelity simulators like GTA-V offers a scalable alternative. However, models trained purely on synthetic data typically suffer from poor generalization when deployed in the real world. This performance drop is primarily attributed to the domain gap: synthetic images possess distinct textures and lighting conditions that do not perfectly match real-world distributions.
The Frequency-Based Domain Gap
Recent research suggests that the domain gap can be analyzed through the lens of image frequency. Convolutional Neural Networks (CNNs) tend to capture global styles (low-frequency information) in early training stages and transition to fine-grained textures (high-frequency information) later. Standard Single Domain Generalization (SDG) methods often focus on only one end of this spectrum.
As shown in Figure 1, both very coarse features (low-frequency) and very fine features (high-frequency) exhibit high domain variancy (DV), while intermediate features are more domain-invariant (DIV). Vanilla training often overfits to these variant features. MRFP aims to expand the model's focus range by perturbing both ends of the spectrum, forcing the network to rely on the robust DIV features.
Multi-Resolution Feature Perturbation (MRFP)
The proposed MRFP framework introduces two primary perturbation modules designed to be used during training and discarded during inference.
High-Resolution Feature Perturbation (HRFP)
HRFP targets the fine-grained, high-frequency textures that often characterize synthetic data. It employs a randomly initialized overcomplete autoencoder. Unlike traditional autoencoders that compress information, an overcomplete structure temporarily increases spatial resolution. This small receptive field ensures that perturbations are localized to textures without altering the global semantic content of the scene.
Style Perturbation (NP+)
To address low-frequency domain variance such as illumination and color shifts, we utilize Normalized Perturbation Plus (NP+). This module perturbs the channel-wise statistics (mean and variance) of feature maps. By simulating a wide variety of styles during training, the model learns to ignore global stylistic artifacts.
MRFP+ and Decoder Perturbation
In the enhanced MRFP+ variant, these perturbations are extended to the decoder layers. This acts as an adversarial challenge for the segmentation head, ensuring that the final classification is robust to feature distortions throughout the entire network hierarchy.
Experimental Results
We evaluated MRFP using synthetic source domains (GTAV and Synthia) and tested generalization across five real-world target domains: Cityscapes, BDD100k, Mapillary, and adverse weather variants (Foggy/Rainy Cityscapes).
Using a ResNet-50 backbone, MRFP+ achieved a mean Intersection over Union (mIoU) of 39.27% across target domains, representing an 8.06% absolute improvement over the vanilla baseline (31.21%). Notably, MRFP+ outperformed state-of-the-art methods like IBN-Net and RobustNet. The method showed particular strength in adverse weather conditions, where standard models often fail due to their reliance on "clean" synthetic textures.
Conclusion
MRFP provides a simple yet effective "plug-and-play" solution for the Sim-2-Real gap in semantic segmentation. By perturbing features at multiple resolutions, it discourages the model from overfitting to domain-specific artifacts. Because the perturbation modules are removed after training, MRFP delivers these generalization benefits with zero impact on inference latency or memory usage, making it ideal for real-time applications like autonomous driving.
Summary updated