Texture-Shape Bias Balancing for Robust Synthetic-to-Real Semantic Segmentation in Automotive NIR Imagery

Felix Stillger, Ben Hamscher, Lukas Hahn, Annika Mütze, Tobias Meisen, Kira MaagarXiv 2026 · arXiv

TL;DR

Training semantic segmentation models for automotive Near-Infrared (NIR) cameras is severely bottlenecked by a lack of annotated real-world data. This paper introduces a generative data augmentation framework that bridges the synthetic-to-real domain gap by combining Target Style Adaptation (TSA) via diffusion models with Voronoi-based Style Diversification (VSD). By explicitly balancing the model's inductive bias between texture and shape, this approach improves real-world segmentation performance by up to 63.6% without requiring a single real-world label.

What Problem Does This Paper Address?

Modern vehicles increasingly rely on computer vision for safety features, ranging from exterior autonomous navigation to interior occupant monitoring (such as driver drowsiness detection mandated by upcoming EU regulations). Near-Infrared (NIR) imaging is ideal for these tasks because it provides stable, high-contrast illumination across day and night cycles without distracting the driver.

However, training deep learning models for semantic segmentation requires massive pixel-level annotated datasets. While synthetic data generated from simulation environments offers a cheap, perfectly labeled alternative, models trained on it suffer severe performance drops when deployed in the real world—a phenomenon known as the domain gap.

This gap is amplified in NIR imaging because most synthetic simulators are designed exclusively for the RGB spectrum. Furthermore, models trained on clean synthetic data develop a severe texture bias, over-relying on local pixel patterns rather than global geometric shapes. When confronted with the completely different texture statistics of real-world NIR sensors, these models break down.

Methods

The authors propose a modular, structure-preserving generative augmentation framework designed to rebalance a model's inductive bias away from texture and toward shape.

Fig. 2: Our method augments data from a source domain, applies a target style adaptation via an LDM, and corrupts the texture by Voronoi style diversification. The resulting augmented dataset is utilized to train a semantic segmentation model which generalizes to real (target) data. Dashed arrows indicate optionally applying only one modular component.

The pipeline consists of two primary components:

  1. Target Style Adaptation (TSA): To translate synthetic RGB images into realistic NIR variants, the authors fine-tune a Latent Diffusion Model (Stable Diffusion XL) using Low-Rank Adaptation (LoRA) on a small, unannotated set of real NIR images. To prevent the diffusion process from destroying small, safety-critical structures (like seatbelts or glasses), they implement a multi-signal conditioning strategy via ControlNet. This incorporates the synthetic segmentation mask, Canny edges from the original image, and Canny edges derived directly from the mask to prevent inter-class boundary bleeding.
  2. Voronoi-based Style Diversification (VSD): To break the model's reliance on local textures, the images are stochastically corrupted using a localized style-transfer technique. The image is partitioned into irregular regions via Voronoi tessellation, and independent artistic styles (sampled from the Painter by Numbers dataset) are applied to each cell using Adaptive Instance Normalization (AdaIN), followed by a grayscale conversion. This forces the downstream segmentation network to ignore local texture consistencies and focus on global geometric boundaries.

Datasets

The framework was validated across both interior and exterior automotive scenarios:

  • Interior Sensing: A proprietary in-cabin dataset consisting of 4,341 synthetic images and 1,151 real-world NIR images. It includes highly detailed, safety-relevant classes such as seat-backrest, seat-belt, child-seat, face, and glasses.
  • Exterior Sensing: The authors paired the synthetic GTA5 dataset (6,766 images) as the source domain with the public RANUS dataset (3,772 real-world RGB-NIR street scene pairs captured in Seoul, South Korea) as the target domain. Both datasets were mapped to 8 shared classes (e.g., road, sky, vegetation, vehicle).

Key Insights and Ideas

  • Texture Bias is a Synth-to-Real Killer: Using a Cue-Decomposition Shape Bias (CDSB) metric, the authors proved that synthetic-only training yields heavily texture-dominated models that fail to generalize to the unique spectral characteristics of NIR imagery.
  • Architecture Dictates Bias Sensitivity: The optimal augmentation strategy depends heavily on the model backbone. Convolutional networks (like DeepLabV3+) are highly texture-biased and benefit most from localized texture diversification (VSD). Conversely, high-capacity transformers (like Mask2Former) possess strong global shape priors from pre-training and perform best under global, uniform stylization.
  • Geometry-Stable Classes Benefit Most: The largest performance gains occurred in classes with stable geometry and clear structural boundaries (e.g., seat backrest and road), confirming that reducing texture bias allows the model to leverage robust shape cues.

Key Results

The combination of TSA and VSD consistently outperformed pure synthetic baselines across multiple architectures (DeepLabV3+, SegFormer, Mask2Former):

  • Exterior Scenes: On the RANUS dataset, the framework closed the domain gap by 63.6% for the CNN-based DeepLabV3+, boosting its median mIoU from a dismal 21.57% to 50.44%.
  • Interior Scenes: The framework closed the domain gap by 28.4% for DeepLabV3+ and 25.0% for SegFormer.
  • Corruption Robustness: Models trained with this pipeline demonstrated significantly higher resilience to severe image distortions—such as uniform noise, low-pass filtering, elastic deformation, and swirl distortions—proving that bias balancing creates fundamentally more robust representations.

Limitations and Future Work

While the framework dramatically improves layout-level and structurally stable classes, fine-grained, small-scale categories (such as glasses or face-mask in the cabin) still present a challenge. The generative diffusion process can occasionally corrupt or omit these high-frequency details, leading to inconsistent performance gains for these specific classes.

For future work, the authors propose embedding this generative pipeline into active self-training or knowledge-distillation frameworks, using the enhanced data to generate highly reliable pseudo-labels for lightweight, deployment-ready models.

Summary updated

Abstract

Semantic segmentation is a fundamental component of visual perception in modern automotive systems, enabling pixel-level scene understanding. Near-Infrared imaging (NIR) offers stable detection under difficult illumination conditions, but the development of domain-specific semantic segmentation models remains challenging due to the lack of high-quality annotated data from real-world scenarios. Synthetic datasets offer a scalable alternative, but models trained on synthetic images often suffer performance degradation when transferred to real domains. We present the first systematic study on synthetic to real domain adaptation for semantic segmentation in NIR images in the automotive domain. We propose a generative augmentation framework that transforms synthetic images into realistic NIR-style variants via our introduced target style adaptation (TSA). TSA fine-tunes a latent diffusion model via low-rank adaptation on a small curated set of real NIR images and applies it to synthetic training data using structure-preserving multi-signal conditioning. To reduce texture bias and improve segmentation robustness, we further apply a Voronoi-based style diversification strategy (VSD) that modifies the original textures while preserving scene geometry. Experiments with multiple model architectures on NIR data from vehicle interiors and street scenes show that balancing inductive bias during training leads to noticeably more robust semantic segmentation and effectively reduces the domain gap in our real-world scenarios by up to 63.6% on exterior and 28.4% on interior data. The code is available at GitHub.
BibTeX
@misc{stillger2026texture,
  author = {Felix Stillger and Ben Hamscher and Lukas Hahn and Annika Mütze and Tobias Meisen and Kira Maag},
  title = {{Texture-Shape Bias Balancing for Robust Synthetic-to-Real Semantic Segmentation in Automotive NIR Imagery}},
  year = {2026},
  eprint = {2606.15072},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.15072},
}

Loading related content...

Figure