TL;DR
Training semantic segmentation models for automotive Near-Infrared (NIR) cameras is severely bottlenecked by a lack of annotated real-world data. This paper introduces a generative data augmentation framework that bridges the synthetic-to-real domain gap by combining Target Style Adaptation (TSA) via diffusion models with Voronoi-based Style Diversification (VSD). By explicitly balancing the model's inductive bias between texture and shape, this approach improves real-world segmentation performance by up to 63.6% without requiring a single real-world label.
What Problem Does This Paper Address?
Modern vehicles increasingly rely on computer vision for safety features, ranging from exterior autonomous navigation to interior occupant monitoring (such as driver drowsiness detection mandated by upcoming EU regulations). Near-Infrared (NIR) imaging is ideal for these tasks because it provides stable, high-contrast illumination across day and night cycles without distracting the driver.
However, training deep learning models for semantic segmentation requires massive pixel-level annotated datasets. While synthetic data generated from simulation environments offers a cheap, perfectly labeled alternative, models trained on it suffer severe performance drops when deployed in the real world—a phenomenon known as the domain gap.
This gap is amplified in NIR imaging because most synthetic simulators are designed exclusively for the RGB spectrum. Furthermore, models trained on clean synthetic data develop a severe texture bias, over-relying on local pixel patterns rather than global geometric shapes. When confronted with the completely different texture statistics of real-world NIR sensors, these models break down.
Methods
The authors propose a modular, structure-preserving generative augmentation framework designed to rebalance a model's inductive bias away from texture and toward shape.
The pipeline consists of two primary components:
- Target Style Adaptation (TSA): To translate synthetic RGB images into realistic NIR variants, the authors fine-tune a Latent Diffusion Model (Stable Diffusion XL) using Low-Rank Adaptation (LoRA) on a small, unannotated set of real NIR images. To prevent the diffusion process from destroying small, safety-critical structures (like seatbelts or glasses), they implement a multi-signal conditioning strategy via ControlNet. This incorporates the synthetic segmentation mask, Canny edges from the original image, and Canny edges derived directly from the mask to prevent inter-class boundary bleeding.
- Voronoi-based Style Diversification (VSD): To break the model's reliance on local textures, the images are stochastically corrupted using a localized style-transfer technique. The image is partitioned into irregular regions via Voronoi tessellation, and independent artistic styles (sampled from the Painter by Numbers dataset) are applied to each cell using Adaptive Instance Normalization (AdaIN), followed by a grayscale conversion. This forces the downstream segmentation network to ignore local texture consistencies and focus on global geometric boundaries.
Datasets
The framework was validated across both interior and exterior automotive scenarios:
- Interior Sensing: A proprietary in-cabin dataset consisting of 4,341 synthetic images and 1,151 real-world NIR images. It includes highly detailed, safety-relevant classes such as
seat-backrest,seat-belt,child-seat,face, andglasses. - Exterior Sensing: The authors paired the synthetic GTA5 dataset (6,766 images) as the source domain with the public RANUS dataset (3,772 real-world RGB-NIR street scene pairs captured in Seoul, South Korea) as the target domain. Both datasets were mapped to 8 shared classes (e.g.,
road,sky,vegetation,vehicle).
Key Insights and Ideas
- Texture Bias is a Synth-to-Real Killer: Using a Cue-Decomposition Shape Bias (CDSB) metric, the authors proved that synthetic-only training yields heavily texture-dominated models that fail to generalize to the unique spectral characteristics of NIR imagery.
- Architecture Dictates Bias Sensitivity: The optimal augmentation strategy depends heavily on the model backbone. Convolutional networks (like DeepLabV3+) are highly texture-biased and benefit most from localized texture diversification (VSD). Conversely, high-capacity transformers (like Mask2Former) possess strong global shape priors from pre-training and perform best under global, uniform stylization.
- Geometry-Stable Classes Benefit Most: The largest performance gains occurred in classes with stable geometry and clear structural boundaries (e.g.,
seat backrestandroad), confirming that reducing texture bias allows the model to leverage robust shape cues.
Key Results
The combination of TSA and VSD consistently outperformed pure synthetic baselines across multiple architectures (DeepLabV3+, SegFormer, Mask2Former):
- Exterior Scenes: On the RANUS dataset, the framework closed the domain gap by 63.6% for the CNN-based DeepLabV3+, boosting its median mIoU from a dismal 21.57% to 50.44%.
- Interior Scenes: The framework closed the domain gap by 28.4% for DeepLabV3+ and 25.0% for SegFormer.
- Corruption Robustness: Models trained with this pipeline demonstrated significantly higher resilience to severe image distortions—such as uniform noise, low-pass filtering, elastic deformation, and swirl distortions—proving that bias balancing creates fundamentally more robust representations.
Limitations and Future Work
While the framework dramatically improves layout-level and structurally stable classes, fine-grained, small-scale categories (such as glasses or face-mask in the cabin) still present a challenge. The generative diffusion process can occasionally corrupt or omit these high-frequency details, leading to inconsistent performance gains for these specific classes.
For future work, the authors propose embedding this generative pipeline into active self-training or knowledge-distillation frameworks, using the enhanced data to generate highly reliable pseudo-labels for lightweight, deployment-ready models.
Summary updated