Sustainable Road Pothole Detection: A Crowdsourcing Based Multi-Sensors Fusion Approach

Hanyu Xin, Yin Ye, Xiaoxiang Na, Huan Hu, Gaoang Wang, Chao Wu, Simon HuSustainability 2023 · DOI

TL;DR

This paper introduces a novel crowdsourcing system for detecting road potholes by fusing data from smartphone accelerometers and in-vehicle cameras. By combining these sensor inputs and employing advanced machine learning techniques, the system achieves improved accuracy and robustness compared to traditional methods, offering a sustainable solution for large-scale road condition monitoring.

What Problem Does This Paper Address?

Maintaining road quality is crucial for safety and cost-efficiency, but traditional methods for detecting road defects like potholes are often slow, expensive, and lack comprehensive coverage. While smartphone sensors (accelerometers) and computer vision offer potential, accelerometer data can be sensitive to driving speed, and image-based detection struggles in varying light conditions. This paper tackles the challenge of creating a scalable, real-time, and cost-effective road pothole detection system that overcomes these limitations by leveraging the ubiquitous nature of smartphones and combining multiple data sources.

Methods

The proposed system employs a multi-sensor fusion approach. Data from smartphone accelerometers and in-vehicle cameras are collected via a dedicated mobile app.

  • Data Preprocessing: Raw accelerometer data is resampled to a consistent frequency (50 Hz) and smoothed using a high-pass filter to remove low-frequency noise from gravity and regular driving maneuvers, retaining high-frequency signals indicative of road anomalies. Video frames are extracted at regular intervals.
  • Feature Extraction: For acceleration data, both time-domain (statistical features like mean, RMS, standard deviation) and frequency-domain features (from FFT, PSD, and autocorrelation) are extracted. Video frames are processed using a pre-trained Image-Net CNN to extract depth features.
  • Detection Modules:
    • Module 1 (Good Light Conditions): Fuses processed accelerometer data and video frame features. Both modalities are encoded into real-valued vectors and projected into a common space. A deep learning model, specifically a Long Short-Term Memory (LSTM) network, is used for anomaly detection.
  • Module 2 (Poor Light Conditions): Relies solely on processed accelerometer data and an LSTM-based recognition model.
  • Multi-Vehicle Fusion and Optimization: Detection results from multiple vehicles are aggregated. A spatial clustering algorithm, specifically OPTICS (Ordering Points To Identify the Clustering Structure), is used to cluster detected anomalies based on spatial density. This helps to filter out noise and synthesize multiple detections into a single, optimized pothole location. The center of each cluster is identified as the final detected pothole location.

Datasets

The study utilized data collected from real-world driving experiments. The dataset consists of:

  • Smartphone accelerometer data (sampled at 100 Hz, then resampled to 50 Hz).
  • Smartphone GPS data (sampled at 1 Hz).
  • In-vehicle video recordings captured by dashcams or onboard cameras.
  • Ground truth labels for road conditions, obtained by manually annotating acceleration data based on onboard video recordings.

The paper does not explicitly mention the creation of a new public benchmark dataset but rather describes the collection and processing of data for their experimental setup.

Key Insights and Ideas

  • Complementary Strengths: The core idea is that accelerometer data (less affected by light) and video data (capturing visual road surface information) have complementary strengths. Fusing them effectively overcomes the individual limitations of each sensor.
  • LSTM for Time-Series Anomaly Detection: The use of LSTM is well-suited for this task because it can effectively learn temporal dependencies in sequential sensor data, crucial for identifying anomalies in road conditions over time.
  • OPTICS for Robust Crowdsourced Data Fusion: OPTICS is a powerful choice for aggregating data from multiple vehicles. Its ability to handle arbitrary cluster shapes and identify noise is vital for dealing with the inherent variability and potential inaccuracies in crowdsourced sensor data.
  • Data Fusion in a Shared Latent Space: Projecting features from different modalities (acceleration and video) into a common "public space" before applying deep learning is a key step that allows the model to learn relationships between them effectively.

Key Results

  • The proposed joint optimization model (fusion of acceleration and video data) achieved the highest accuracy on the testing set, reaching 0.965.
  • The joint optimization model also demonstrated superior performance in Precision (0.893), Recall (0.821), and F1 Score (0.856) compared to individual classifiers like SVM, RF, and LSTM.
  • The paper states that the proposed method can improve accuracy by 6% compared to traditional methods (though the specific traditional method for this comparison isn't explicitly detailed in the abstract, it's implied to be a baseline or single-sensor approach).
  • Experiments showed that the detection accuracy was largely unaffected by vehicle type and speed when speeds exceeded 20 m/s, indicating robustness.
  • The study also found that while phone placement within the car could slightly affect raw signal amplitude, the preprocessed data and the proposed method were not sensitive to phone position, demonstrating suitability for crowdsourcing.

Limitations and Future Work

  • Environmental Dependency: While the fusion approach mitigates the impact of ambient light on image-based detection, the system still has two modules, with one relying solely on acceleration in poor light. This suggests that performance might still be suboptimal in extremely challenging conditions (e.g., heavy rain, fog, complete darkness) where visual data is entirely unusable and acceleration signals might be obscured by other vibrations.
  • Data Imbalance: The paper acknowledges that road anomaly detection is a highly unbalanced classification problem (more normal road segments than potholes). While SMOTE was used for data augmentation, addressing this imbalance remains a challenge for achieving consistently high detection rates for anomalies.
  • Real-time Implementation: The study mentions that the system was operated offline during research, but the designed pipeline can be migrated online for practical use. This suggests that real-time deployment and its associated computational challenges are a future consideration.
  • Crowdsourcing Infrastructure: While the system leverages crowdsourcing, the paper doesn't delve deeply into the complexities of managing and scaling such a system in a real-world, large-scale deployment, including user incentives, data quality control, and network infrastructure.

Summary updated

Abstract

Real-time road quality monitoring, involves using technologies to collect data on the conditions of the road, including information on potholes, cracks, and other defects. This information can help to improve safety for drivers and reduce costs associated with road damage. Traditional methods are time-consuming and expensive, leading to limited spatial coverage and delayed responses to road conditions. With the widespread use of smartphones and ubiquitous computing technologies, data can be collected from built-in sensors of mobile phones and in-vehicle video, on a large scale. This has raised the question of how these data can be used for road pothole detection and has significant practical relevance. Current methods either use acceleration sequence classification techniques, or image recognition techniques based on deep learning. However, accelerometer-based detection has limited coverage and is sensitive to the driving speed, while image recognition-based detection is highly affected by ambient light. To address these issues, this study proposes a method that utilizes the fusion of accelerometer data and in-vehicle video data, which is uploaded by the participating users. The preprocessed accelerometer data and intercepted video frames, were then encoded into real-valued vectors, and projected into the public space. A deep learning-based training approach was used to learn from the public space and identify road anomalies. Spatial density-based clustering was implemented in a multi-vehicle scenario, to improve reliability and optimize detection results. The performance of the model is evaluated with confusion matrix-based classification metrics. Real-world vehicle experiments are carried out, and the results demonstrate that the proposed method can improve accuracy by 6% compared to the traditional method. Consequently, the proposed method provides a novel approach for large-scale pavement anomaly detection.
BibTeX
@inproceedings{xin2023sustainable,
  author = {Hanyu Xin and Yin Ye and Xiaoxiang Na and Huan Hu and Gaoang Wang and Chao Wu and Simon Hu},
  title = {{Sustainable Road Pothole Detection: A Crowdsourcing Based Multi-Sensors Fusion Approach}},
  booktitle = {Sustainability},
  year = {2023},
  doi = {10.3390/su15086610},
}
Figure