Abstract
While deep neural networks for environment perception tasks in autonomous driving systems often achieve impressive performance on clean and well-prepared images, their robustness under real conditions, i.e., on images being perturbed with noise patterns or adversarial attacks, is often subject to a significantly decreased performance. In this chapter, we address this problem for the task of semantic segmentation by proposing multi-task training with the additional task of depth estimation with the goal to improve the DNN robustness. This method has a very wide potential applicability as the additional depth estimation task can be trained in a self-supervised fashion, relying only on unlabeled image sequences during training. The final trained segmentation DNN is, however, still applicable on a single-image basis during inference without additional computational overhead compared to the single-task model. Additionally, our evaluation introduces a measure which allows for a meaningful comparison between different noise and attack types. We show the effectiveness of our approach on the Cityscapes and KITTI datasets, where our method improves the DNN performance w.r.t. the single-task baseline in terms of robustness against multiple noise and adversarial attack types, which is supplemented by an improved absolute prediction performance of the resulting DNN.
Introduction
Deep Neural Networks (DNNs) serve as the primary perception systems for autonomous vehicles, handling critical tasks such as semantic segmentation—the pixel-wise classification of a scene into categories like "road," "pedestrian," or "car." Despite their high accuracy on standard benchmarks, these models are notoriously sensitive to perturbations. In real-world deployment, sensor noise, weather conditions, or intentional adversarial attacks can lead to catastrophic misclassifications.
Existing defense mechanisms often involve trade-offs, such as requiring expensive manual labeling, increasing computational latency during inference, or degrading performance on clean data. This research proposes a multi-task learning (MTL) framework that hardens models against noise without these drawbacks.
Methodology: Multi-Task Learning with Self-Supervision
The core innovation lies in training a shared feature extractor (encoder) to perform two tasks simultaneously: a primary supervised task and an auxiliary self-supervised task.
Architecture and Training Pipeline
The model consists of a shared encoder and two task-specific decoders.
- Primary Task: Supervised semantic segmentation using standard ground-truth labels.
- Auxiliary Task: Self-supervised monocular depth estimation.
The depth estimation task is "self-supervised" because it utilizes the geometric consistency between sequential video frames rather than manual labels. By learning to reconstruct frames, the model develops a fundamental understanding of 3D scene structure. To balance the learning signals from these disparate tasks, the authors employ the GradNorm algorithm, which dynamically adjusts task weights during training to prevent one gradient from dominating the other.
Inference Efficiency
A significant advantage of this approach is its efficiency. During real-world deployment (inference), the depth decoder is discarded. The car only runs the segmentation task, ensuring that the robustness gains come with zero additional computational cost compared to a standard single-task model.
Experimental Results and Insights
The framework was evaluated on the Cityscapes and KITTI datasets. Robustness was tested against various perturbations, including Gaussian noise, Salt and Pepper noise, and adversarial attacks like the Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD).
- Performance Boost: On clean Cityscapes data, the multi-task model improved the mean Intersection-over-Union (mIoU) from 63.5% to 67.9%.
- Enhanced Robustness: Under significant Gaussian noise, the multi-task model retained 33.3% of its original performance, nearly doubling the relative robustness of the single-task baseline (18.4%).
- Cross-Domain Benefits: Surprisingly, training the auxiliary depth task on a different dataset (e.g., KITTI) while training segmentation on Cityscapes yielded superior robustness. This suggests that exposure to diverse environmental geometries helps the encoder ignore irrelevant 2D noise.
Conclusion
By integrating self-supervised depth estimation as an auxiliary task, this research demonstrates that DNNs can achieve both higher accuracy and greater robustness against noise and adversarial attacks. The method's reliance on unlabeled video data for the auxiliary task and its zero-cost inference make it a highly practical solution for safety-critical autonomous driving systems.
Summary updated