Vision-based Navigation of Autonomous Vehicle in Roadway Environments with Unexpected Hazards

Mhafuzul Islam, Mashrur Chowdhury, Hongda Li, Hongxin HuarXiv 2018 · arXiv

Abstract

Traditional end-to-end deep learning models for autonomous driving often struggle with "out-of-distribution" hazards, such as unexpected roadway debris. This paper introduces a modular hazard-aware navigation system that utilizes real-time semantic segmentation and a calculated "threat factor" to improve obstacle avoidance. By explicitly identifying hazards and fusing their spatial context into the driving model, the proposed architecture achieves a 21% improvement in steering accuracy over standard models in simulated environments.

Introduction

Most current autonomous driving systems rely on end-to-end Deep Neural Networks (DNNs) that map camera images directly to steering commands. While efficient, these "black box" models are notoriously brittle: if they encounter an object not well-represented in their training data—such as a fallen crate or a stray oil drum—they may fail to recognize it as a hazard, leading to potentially fatal collisions. This research seeks to bridge this gap by creating a pipeline that detects, analyzes, and reacts to unexpected roadway objects in real-time.

Methodology

The authors propose a three-stage modular DNN architecture designed to provide an explicit "understanding" of roadway hazards.

The proposed three-stage modular DNN architecture for hazard-aware navigation.

1. Hazard Detection and Segmentation

The first stage identifies hazardous objects and generates a pixel-level "segmented map" of the threat. This is achieved using a Fully Convolutional Network (FCN) based on the VGGNet architecture.

The FCN-Network architecture used for pixel-wise hazard object segmentation.

2. Hazard Analysis and Avoidance

This stage calculates a Threat Factor ($T_f$)—a value between 0 and 1 based on the object's proximity and position. The system then "fuses" the original camera image with the segmented hazard map, weighted by this threat factor. This process effectively "highlights" the danger for the subsequent driving model. Two procedures were tested: one using Radar for distance and another using vision-only pixel calculations.

3. Driving Model

A CNN (similar to NVIDIA’s DAVE-2) takes the fused image and predicts the optimal steering angle to navigate safely around the hazard.

Experimental Setup

Due to a lack of existing datasets combining steering angles with ground-truth hazard labels, the researchers developed a custom dataset using the Webots robotics simulation platform.

  • Environment: A 1663m roadway featuring 16 curves and two intersections.
  • Hazards: Various objects including rocks, wooden boxes, oil barrels, wooden pallets, and pipe sections.
  • Data Augmentation: The initial 1,390 images were augmented to 2,780 images through rotation, brightness adjustments, and horizontal flipping.

Results and Discussion

The system was evaluated across three cases: a standard DNN (Case 1), a Radar-assisted segmentation model (Case 2), and a vision-only segmentation model (Case 3).

  • Safety Performance: Case 2 improved steering accuracy by 21% over the standard model, while the vision-only Case 3 improved it by 18%. This suggests that high-cost sensors like Radar might not be strictly necessary if segmentation is robust.
  • Trajectory Precision: In trajectory tests, the standard model (Case 1) exhibited a Root Mean Square Error (RMSE) of 0.52, whereas the hazard-aware Case 2 achieved a significantly lower 0.07.
  • Explicit Learning: Unlike traditional models that learn safety implicitly, this approach uses explicit hazard features, ensuring the model recognizes specific pixels as threats.

Conclusion

The proposed modular architecture demonstrates that integrating semantic context significantly enhances the safety of autonomous navigation. While currently limited to simulated environments, future work will address real-world variables such as weather and motion blur, as well as temporal fusion to provide the model with a "memory" of hazards across frames.

Summary updated

Abstract

Vision-based navigation of autonomous vehicles primarily depends on the Deep Neural Network (DNN) based systems in which the controller obtains input from sensors/detectors, such as cameras and produces a vehicle control output, such as a steering wheel angle to navigate the vehicle safely in a roadway traffic environment. Typically, these DNN-based systems of the autonomous vehicle are trained through supervised learning; however, recent studies show that a trained DNN-based system can be compromised by perturbation or adversarial inputs. Similarly, this perturbation can be introduced into the DNN-based systems of autonomous vehicle by unexpected roadway hazards, such as debris and roadblocks. In this study, we first introduce a roadway hazardous environment (both intentional and unintentional roadway hazards) that can compromise the DNN-based navigational system of an autonomous vehicle, and produces an incorrect steering wheel angle, which can cause crashes resulting in fatality and injury. Then, we develop a DNN-based autonomous vehicle driving system using object detection and semantic segmentation to mitigate the adverse effect of this type of hazardous environment, which helps the autonomous vehicle to navigate safely around such hazards. We find that our developed DNN-based autonomous vehicle driving system including hazardous object detection and semantic segmentation improves the navigational ability of an autonomous vehicle to avoid a potential hazard by 21% compared to the traditional DNN-based autonomous vehicle driving system.
BibTeX
@misc{islam2018vision,
  author = {Mhafuzul Islam and Mashrur Chowdhury and Hongda Li and Hongxin Hu},
  title = {{Vision-based Navigation of Autonomous Vehicle in Roadway Environments with Unexpected Hazards}},
  year = {2018},
  eprint = {1810.03967},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1810.03967},
}
Figure