Robust Semantic Segmentation by Redundant Networks With a Layer-Specific Loss Contribution and Majority Vote

Andreas Bär, Marvin Klingner, Serin Varghese, Fabian Hüger, Peter Schlicht, Tim Fingscheidt

Introduction

Deep Neural Networks (DNNs) are the backbone of modern computer vision, but they possess a significant vulnerability: adversarial examples (AEs). These are inputs modified by tiny, often invisible perturbations that cause a model to make confident, catastrophic errors—such as mistaking a pedestrian for a stretch of empty road. In safety-critical applications like autonomous driving, this lack of robustness is a major hurdle.

Previous attempts to harden models against AEs often resulted in a "robustness tax," where the model became significantly worse at processing normal, "clean" images. This paper introduces a redundant "watchdog" system that maintains high accuracy on clean data while providing a robust defense against adversarial attacks.

System Architecture

The proposed framework utilizes a triple-redundancy architecture consisting of three distinct networks:

  1. Static Teacher (T): A fixed, high-performance model providing baseline semantic labels.
  2. Static Student (S): A fixed model with a different architecture or initialization.
  3. Adaptive Student (A): A model that continuously learns "online" while the vehicle is operating, acting as a moving target for attackers.

Methods: Layer-Specific IFM Loss

The core innovation of this work is the training objective for the Adaptive Student (A). It is optimized using two competing goals:

  • Knowledge Distillation: It attempts to match the semantic label outputs ($y^T$) of the Static Teacher using Cross-Entropy ($J^{CE}$) loss.
  • Inverse Feature Matching (IFM): It uses a novel IFM loss ($J^{IFM'}$) to force its internal features ($f_\ell^A$) to be dissimilar from the Static Student’s features ($f_\ell^S$).
Conceptual diagram of the Adaptive Student's dual objectives: matching the teacher's soft label output while extracting dissimilar internal features from the static student.

Crucially, the authors found that applying this divergence to every layer confuses the model and ruins clean-data performance. By applying the IFM loss only to specific layers (such as the second downsampling layer), the system targets the specific paths through which adversarial errors propagate without destroying the model's fundamental understanding of the scene.

Training and Online Operation

The system operates in distinct phases to ensure both initial accuracy and operational resilience.

The complete system lifecycle, illustrating supervised pre-training, parameter freezing, online adaptation via teacher distillation/IFM loss, and final majority-vote inference.

During the Online Vehicle Operation, the Adaptive Student is fine-tuned using unlabeled camera data ($x$). Because its weights are constantly changing, it becomes a "moving target" that is difficult for an attacker to model. During inference, the system compares the outputs of all three networks. If the static networks are compromised by an attack, their outputs will diverge from the Adaptive Student. A majority vote or distance-based "watchdog" mechanism then selects the most reliable output, effectively bypassing the attacked component.

Key Results

The researchers evaluated their approach using the Cityscapes dataset, a standard benchmark for urban semantic segmentation. The results highlight the effectiveness of the layer-specific approach:

  • Clean Data Stability: The system maintained an mIoU of ~75.7% on clean data, nearly matching undefended state-of-the-art models.
  • Massive Robustness Gains: Under strong adversarial attacks, standard networks saw performance crater to 1–2% mIoU. The proposed redundant system maintained roughly 50% mIoU under the same conditions.
  • Improved Adaptation: The layer-specific loss made the adaptive student 20% better (absolute mIoU) than previous redundant frameworks that utilized global feature matching losses.

Conclusion

By treating vision as an ensemble problem with a dynamic "watchdog," this framework provides a viable path toward resilient autonomous systems. The combination of online adaptation and strategic, layer-specific feature divergence allows the system to remain accurate during everyday driving while successfully identifying and mitigating adversarial interference.

Summary updated

Abstract

The lack of robustness shown by deep neural networks (DNNs) questions their deployment in safety-critical tasks, such as autonomous driving. We pick up the recently introduced redundant teacher-student frameworks (3 DNNs) and propose in this work a novel error detection and correction scheme with application to semantic segmentation. It obtains its robustnesss by an online-adapted and therefore hard-to-attack student DNN during vehicle operation, which builds upon a novel layer-dependent inverse feature matching (IFM) loss. We conduct experiments on the Cityscapes dataset showing that this loss renders the adaptive student to be more than 20% absolute mean intersection-over-union (mIoU) better than in previous works. Moreover, the entire error correction virtually always delivers the performance of the best non-attacked network, resulting in an mIoU of about 50% even under strongest attacks (instead of 1...2%), while keeping the performance on clean data at about original level (ca. 75.7%).
BibTeX
@inproceedings{bar2020robust,
  author = {Andreas Bär and Marvin Klingner and Serin Varghese and Fabian Hüger and Peter Schlicht and Tim Fingscheidt},
  title = {{Robust Semantic Segmentation by Redundant Networks With a Layer-Specific Loss Contribution and Majority Vote}},
  booktitle = {Workshops @ CVPR},
  year = {2020},
  doi = {10.1109/cvprw50498.2020.00174},
}
Figure