Reimagine BiSeNet for Real-Time Domain Adaptation in Semantic Segmentation

Antonio Tavera, Carlo Masone, Barbara CaputoarXiv 2021 · arXiv

Introduction

Semantic segmentation—the process of assigning a class label to every pixel in an image—is a cornerstone of autonomous driving systems. However, deploying these models in real-world vehicles faces two significant challenges: the resource gap and the domain gap. High-performance models often require computational power beyond the reach of embedded devices, and models trained on cost-effective synthetic data (like video games) frequently fail when exposed to the nuances of real-world streets.

Researchers from Politecnico di Torino address these hurdles by reimagining the Bilateral Segmentation Network (BiSeNet) for Unsupervised Domain Adaptation (UDA). By introducing a novel "Light&Thin" discriminator, they demonstrate that real-time adaptation is not only possible but can be achieved with a fraction of the usual computational cost.

Methodology: The Light&Thin Approach

The proposed framework utilizes Adversarial Training to bridge the domain gap. In this setup, the BiSeNet model acts as a generator, producing feature maps from both synthetic (source) and real (target) images. A discriminator network then attempts to distinguish between these domains. To fool the discriminator, the generator learns to produce domain-invariant features, effectively "translating" its knowledge from simulation to reality.

The proposed architecture integrating BiSeNet with the novel Light&Thin Discriminator for adversarial domain adaptation.

The core innovation lies in the Light&Thin discriminator design:

  • Depthwise Separable Convolutions: By splitting spatial and channel-wise processing, the model significantly reduces the number of parameters and operations.
  • Shallow Architecture: Unlike standard discriminators that use five or more layers, this version uses only three, reaching local optima faster and more effectively for the binary classification task of domain detection.
  • Efficiency: This design is specifically tailored for deployment on resource-constrained embedded hardware.

Experimental Results

The team evaluated the framework using standard synthetic-to-real benchmarks, transitioning from GTA5 and SYNTHIA datasets to the real-world Cityscapes dataset.

Quantitative Performance

The "Light&Thin" discriminator (FCD-LightThin) consistently outperformed the standard, heavier Fully Convolutional Discriminator (FCD) across multiple benchmarks. For the GTA5 to Cityscapes task, the Mean Intersection over Union (mIoU) improved from 31.89% to 33.22%.

Quantitative results for the GTA5 to Cityscapes domain adaptation task, showing the performance of the Light&Thin model across various object classes.

Efficiency Gains

The reduction in computational overhead is the most striking result:

  • Parameter Reduction: The standard discriminator contains approximately 2.78 million parameters, while the Light&Thin version contains only 13,316—a 200x reduction.
  • FLOPs Reduction: Floating Point Operations (FLOPs) dropped from 30.88G to 1.03G, making the adaptation process significantly more viable for real-time applications.

Conclusion and Future Work

This research proves that bridging the domain gap does not require massive, server-grade GPUs. By optimizing the discriminator architecture, the authors achieved better segmentation accuracy on real-world data while maintaining a negligible impact on inference time.

While a performance gap remains between adapted models and those trained directly on real-world data (33.22% vs. 64.17% mIoU), this work provides a scalable foundation for real-time UDA. Future efforts will focus on applying these lightweight principles to even more powerful segmentation backbones and further narrowing the synthetic-to-real performance gap.

Summary updated

Abstract

Semantic segmentation models have reached remarkable performance across various tasks. However, this performance is achieved with extremely large models, using powerful computational resources and without considering training and inference time. Real-world applications, on the other hand, necessitate models with minimal memory demands, efficient inference speed, and executable with low-resources embedded devices, such as self-driving vehicles. In this paper, we look at the challenge of real-time semantic segmentation across domains, and we train a model to act appropriately on real-world data even though it was trained on a synthetic realm. We employ a new lightweight and shallow discriminator that was specifically created for this purpose. To the best of our knowledge, we are the first to present a real-time adversarial approach for assessing the domain adaption problem in semantic segmentation. We tested our framework in the two standard protocol: GTA5 to Cityscapes and SYNTHIA to Cityscapes. Code is available at: https://github.com/taveraantonio/RTDA.
BibTeX
@misc{tavera2021reimagine,
  author = {Antonio Tavera and Carlo Masone and Barbara Caputo},
  title = {{Reimagine BiSeNet for Real-Time Domain Adaptation in Semantic Segmentation}},
  year = {2021},
  eprint = {2110.11662},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2110.11662},
}
Figure