Weakly Supervised Patch Label Inference Networks for Efficient Pavement Distress Detection and Recognition in the Wild

Sheng Huang, Wenhao Tang, Guixin Huang, Luwen Huangfu, Dan YangIEEE Trans. ITS 2022 · DOI · OpenReview

TL;DR

This paper introduces WSPLIN, a novel end-to-end deep learning framework for automatically detecting and recognizing pavement distresses in road images. By transforming the problem into a weakly supervised patch-level inference task, WSPLIN efficiently handles high-resolution images with small distress areas, outperforming existing methods in both accuracy and speed.

What Problem Does This Paper Address?

Pavement maintenance is crucial for road safety and longevity, and automated distress detection and recognition are key to efficient management. However, existing deep learning methods often struggle with the unique characteristics of pavement images: their high resolution and the fact that distresses (like cracks) typically occupy a very small portion of the image. This makes it difficult for models to learn effectively. Furthermore, many current approaches are not end-to-end trainable, requiring complex multi-stage processes and limiting their practical application. The paper aims to bridge this gap by proposing a more efficient and integrated solution.

Methods

The core of WSPLIN is its innovative approach to handling pavement images. Instead of treating the entire image as a single input, WSPLIN first divides the image into smaller patches. This is done using three different strategies:

  • Sliding Window (SW): A standard approach that creates overlapping patches.
  • Image Pyramid (IP): This strategy processes the image at multiple scales (resolutions), allowing the model to capture both fine details and broader context. This is the default and best-performing version.
  • Sparse Sampling (SS): A lightweight version that intelligently selects a subset of patches from the image pyramid to reduce computational load and speed up training, offering a good trade-off between performance and efficiency.

These patches are then fed into a Patch Label Inference Network (PLIN), which learns to infer "pseudo-labels" for each patch. Crucially, this inference is done in a weakly supervised manner, meaning it doesn't require pixel-level annotations for the patches. To guide this process, two key components are introduced:

  • Patch Label Sparsity Constraint (PLSC): This constraint leverages the prior knowledge that most pavement patches are normal (no distress). It encourages the PLIN to output low confidence scores for normal patches, forcing it to focus on identifying distresses.
  • Comprehensive Decision Network (CDN): This network takes the inferred patch labels from PLIN and aggregates them to produce the final image-level prediction for distress detection and recognition. The CDN helps to guide the training of PLIN in an end-to-end fashion.

This end-to-end architecture, combined with the patch-level inference and sparsity constraint, allows WSPLIN to effectively learn from image-level labels while still exploiting local patch information and multi-scale context.

Datasets

The paper evaluates WSPLIN on two datasets:

  • CQU-BPDD: A large-scale bituminous pavement distress dataset designed for pavement image classification, featuring seven types of distress. This dataset is used for evaluating the model across various application settings.
  • Crack500-PDD: A newly constructed, smaller-scale dataset augmented from the well-known Crack500 dataset. It specifically focuses on crack detection and includes both distressed and normal pavement images.

Key Insights and Ideas

  • Patch-level inference is key for high-resolution, low-distress-ratio images: Breaking down large images into patches allows the model to focus on the small, critical areas where distress occurs, overcoming the limitations of global image analysis.
  • Weak supervision at the patch level is sufficient: By using a sparsity constraint and a comprehensive decision network, the model can learn meaningful patch labels without requiring expensive pixel-level annotations.
  • Multi-scale processing (Image Pyramid) significantly improves performance: Capturing information at different resolutions helps the model to better understand the context and scale of various distress types.
  • End-to-end training is crucial for efficiency and integration: Combining detection and recognition into a single, trainable framework simplifies the pipeline and improves overall performance.
  • The "normal" class is a critical prior: Explicitly modeling the fact that most pavement images are normal is a powerful way to guide the model's learning process.

Key Results

  • Superior Performance: WSPLIN consistently outperforms existing methods, including IOPLIN (a previous state-of-the-art patch-based method), on both pavement distress detection and recognition tasks across various settings. For instance, on the CQU-BPDD dataset, WSPLIN-IP shows significant gains in F1 scores compared to baselines.
  • Efficiency Gains: WSPLIN-SS, a lightweight version, achieves training times that are up to 74% faster than IOPLIN, while maintaining comparable performance. WSPLIN-IP also shows improved training efficiency over IOPLIN.
  • Interpretability: The patch labels produced by PLIN offer interpretable intermediate results, providing insights into the rough location and type of distress.
  • Robustness: The model demonstrates good performance even with imbalanced class distributions, as indicated by the F1 scores.

Limitations and Future Work

  • Limitations: While WSPLIN shows excellent results, the paper notes that the Crack500-PDD dataset, being primarily designed for crack detection, might be too simple for fully evaluating pavement distress classification methods. The performance of WSPLIN-SW, while decent, is slightly lower than IOPLIN, suggesting that scale variation is indeed an important factor.
  • Future Work: The authors suggest exploring the integration of Spatial Transformer Networks (STN) to further enhance defect classification capabilities. Additionally, they propose investigating self-supervised learning methods for pre-training models, which could potentially lead to even better performance.

Summary updated

Abstract

Automatic image-based pavement distress detection and recognition are vital for pavement maintenance and management. However, existing deep learning-based methods largely omit the specific characteristics of pavement images, such as high image resolution and low distress area ratio, and are not end-to-end trainable. In this paper, we present a series of simple yet effective end-to-end deep learning approaches named Weakly Supervised Patch Label Inference Networks (WSPLIN) for efficiently addressing these tasks under various application settings. WSPLIN transforms the fully supervised pavement image classification problem into a weakly supervised pavement patch classification problem for solutions. Specifically, WSPLIN first divides the pavement image under different scales into patches with different collection strategies and then employs a Patch Label Inference Network (PLIN) to infer the labels of these patches to fully exploit the resolution and scale information. Notably, we design a patch label sparsity constraint based on the prior knowledge of distress distribution and leverage the Comprehensive Decision Network (CDN) to guide the training of PLIN in a weakly supervised way. Therefore, the patch labels produced by PLIN provide interpretable intermediate information, such as the rough location and the type of distress. We evaluate our method on a large-scale bituminous pavement distress dataset named CQU-BPDD and the augmented Crack500 (Crack500-PDD) dataset, which is a newly constructed pavement distress detection dataset augmented from the Crack500. Extensive results demonstrate the superiority of our method over baselines in both performance and efficiency. The source codes of WSPLIN are released on https://github.com/DearCaat/wsplin.
BibTeX
@article{huang2022weakly,
  author = {Sheng Huang and Wenhao Tang and Guixin Huang and Luwen Huangfu and Dan Yang},
  title = {{Weakly Supervised Patch Label Inference Networks for Efficient Pavement Distress Detection and Recognition in the Wild}},
  journal = {IEEE Trans. ITS},
  year = {2022},
  doi = {10.1109/tits.2023.3245192},
}
Figure