Automatic Obstacle Detection Method for the Train Based on Deep Learning

Qiang Zhang, Fei Yan, Weina Song, Rui Wang, Gen LiSustainability 2023 · DOI

TL;DR

This paper presents a deep learning-based system for automatic obstacle detection on train tracks, crucial for enhancing the safety of fully automated train operations. By fusing visual and LiDAR data, the system accurately identifies rail regions and potential obstacles, even in challenging weather conditions, and determines if they pose a risk to the train.

What Problem Does This Paper Address?

Modern trains, especially those operating in fully automatic modes (FAO), rely heavily on signaling systems but often lack robust environmental perception capabilities. This gap poses a significant safety risk, as unexpected obstacles on the tracks (e.g., pedestrians, debris, or other vehicles) can lead to collisions. Existing systems may be affected by environmental factors like poor lighting or weather, or they might not accurately distinguish between obstacles on the track and those nearby. The paper aims to address this by developing an intelligent system that can reliably detect obstacles in all weather conditions and assess their proximity to the train's path.

Methods

The proposed system employs a multi-stage deep learning approach:

  • Rail Region Identification: This module uses a convolutional neural network (CNN) with a semantic segmentation architecture (inspired by SegNet and ERFNet) to precisely identify the rail track area in images. This involves cascading downsampling, convolution, dilated convolution (to capture multi-scale features), and deconvolution for upsampling, culminating in a Softmax layer for pixel-wise classification.
  • Obstacle Detection: Leveraging the SSD (Single Shot MultiBox Detector) framework, this component uses a CNN with depthwise separable convolutions and residual connections for efficient feature extraction. It performs bounding box regression and obstacle classification to identify potential hazards.
  • Visual-LiDAR Fusion: This is a key innovation. The system fuses data from industrial cameras and LiDAR sensors. It uses a post-fusion approach, meaning individual sensor data is processed first, and then the results are combined. This fusion aims to overcome the limitations of each sensor, with LiDAR providing depth information that cameras lack and cameras providing richer visual detail for obstacle classification.
  • Spatial Relationship Judgment: After detecting obstacles and the rail region, this module determines if an obstacle is actually on the train's path. It analyzes the geometric relationship between the detected obstacle and the rail track centerline, considering factors like track curvature and potential turnouts, to decide whether to trigger a warning.

Datasets

The paper describes experiments conducted on the Hong Kong Tsuen Wan metro line. While specific datasets for training are not explicitly detailed, the system was trained using data collected from this operational line, with video resolutions of 1280x720. The evaluation involved over 200,000 km of cumulative mileage and 50,000 km of collected data.

Key Insights and Ideas

  • Synergistic Fusion: The combination of visual and LiDAR data is crucial. LiDAR provides accurate depth, while cameras offer rich visual context, allowing for more robust detection and classification of obstacles, especially in adverse conditions where one sensor might fail.
  • Contextual Obstacle Assessment: Simply detecting an obstacle isn't enough. The system's ability to determine if an obstacle is on the track and poses an immediate threat, rather than just being nearby, is a critical safety enhancement.
  • Deep Learning for Rail Specifics: Adapting state-of-the-art deep learning techniques like semantic segmentation and object detection to the specific challenges of rail environments (e.g., track geometry, varying lighting) is key to achieving high performance.
  • All-Weather Capability: The fusion approach significantly improves the system's reliability across different weather and lighting conditions, a major limitation for vision-only systems.

Key Results

The system demonstrated strong performance on the Hong Kong Tsuen Wan line:

  • Perception Rate (Precision): 99.994%
  • Recall Rate: 100%
  • Detection Range:
    • Straight tracks: > 240 meters
    • 3% grade ramps: > 100 meters
    • Sharpest curve (R=312m): > 70 meters
  • False Alarm Rate (FAR): 0.008%
  • Miss Detection Rate (MDR): 0%

The paper highlights that a miss detection rate of 0 is critical for safety, and their system achieves this, indicating it meets practical application requirements.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while false alarms are minimal and don't compromise safety, they still exist.
  • Neural networks, in general, can have uncertainty, especially in dynamic environments.
  • The current system focuses on on-board obstacle detection, but the braking distance of trains can be long, and perception distance might not always meet braking requirements in curves.
  • Future Work:
    • Develop vehicle-road collaborative perception and improve obstacle detection distance.
  • Investigate methods to estimate and improve the interpretability of the model's uncertainty.
  • Further research into high RAMS (Reliability, Availability, Maintainability, Safety) systems for future driverless operations.

Summary updated

Abstract

Automatic obstacle detection is of great significance for improving the safety of train operation. However, the existing autonomous operation of trains mainly depends on the signaling control system and lacks the extra equipment to perceive the environment. To further enhance the efficiency and safety of the widely deployed fully automatic operation (FAO) systems of the train, this study proposes an intelligent obstacle detection system based on deep learning. It collects perceptual information from industrial cameras and light detection and ranging (LiDAR), and mainly implements the functionality including rail region detection, obstacle detection, and visual–LiDAR fusion. Specifically, the first two parts adopt deep convolutional neural network (CNN) algorithms for semantic segmentation and object detection to pixel-wisely identify the rail track area ahead and detect the potential obstacles on the rail track, respectively. The visual–LiDAR fusion part integrates the visual data with the LiDAR data to achieve environmental perception for all weather conditions. It can also determine the geometric relationship between the rail track and obstacles to decide whether to trigger a warning alarm. Experimental results show that the system proposed in this study has strong performance and robustness. The system perception rate (precision) is 99.994% and the recall rate reaches 100%. The system, applied to the metro Hong Kong Tsuen Wan line, effectively improves the safety of urban rail train operation.
BibTeX
@inproceedings{zhang2023automatic,
  author = {Qiang Zhang and Fei Yan and Weina Song and Rui Wang and Gen Li},
  title = {{Automatic Obstacle Detection Method for the Train Based on Deep Learning}},
  booktitle = {Sustainability},
  year = {2023},
  doi = {10.3390/su15021184},
}
Figure