TL;DR
This paper introduces TB-Net, a novel three-stream deep learning architecture designed for fine-grained segmentation of pavement diseases and landmarks. By effectively fusing spatial, contextual, and boundary information, TB-Net achieves state-of-the-art performance on a challenging new dataset, paving the way for more accurate and comprehensive road infrastructure inspection.
What Problem Does This Paper Address?
Maintaining road infrastructure like bridges and airports is crucial for public safety, and accurate pavement condition assessment is a key part of this. While existing methods primarily focus on detecting and segmenting long, thin cracks, real-world pavement issues are far more diverse. These include various types of diseases with irregular shapes and sizes, as well as important landmarks like markings and lights. Segmenting these diverse elements accurately is challenging due to their varied appearances, sizes, and the inherent noise and illumination variations in pavement images. This paper aims to bridge this gap by developing a single model capable of fine-grained segmentation of a wide range of pavement diseases and typical objects.
Methods
The core of this work is the TB-Net (Three-Stream Boundary-Aware Network). This architecture is designed to leverage three distinct types of information:
- Spatial Stream: This stream focuses on capturing rich, high-resolution spatial features from the input image. It uses a shallow convolutional structure with small strides to preserve spatial detail.
- Context Stream: This stream aims to model contextual relationships over local features. It employs a Context-Aware Attention (CAA) mechanism, built upon features extracted from a pre-trained ResNet-101, to capture global dependencies and enhance feature representations.
- Boundary Stream: This stream is specifically designed to learn detailed boundary information, crucial for accurate segmentation of thin or small objects. It utilizes a Global-Gated Convolution (GGC) module, inspired by gated convolutions, to highlight boundary-relevant features and filter out noise.
These three streams are fused in a Feature Fusion module. The network is trained using a dual-task loss function that jointly optimizes for both segmentation and boundary prediction. This loss function includes a weighted cross-entropy loss for segmentation to handle class imbalance and a binary cross-entropy loss for boundary prediction.
Datasets
The authors introduce a newly collected airport pavement disease dataset for training and evaluation. This dataset contains 3946 images captured with a gray-scale camera, featuring complex cement/asphalt pavements under varying illumination. It includes 3171 images for training, 793 for testing, and 20 for validation. The dataset annotates eight semantic classes: Crack, Cornerfracture, Seambroken, Patch, Repair, Slab, Track (which includes markings and stains), and Light (ground lighting equipment), plus a background class.
Key Insights and Ideas
- Multi-stream fusion is key: The paper demonstrates that effectively combining spatial, contextual, and boundary information is crucial for achieving high performance in fine-grained pavement segmentation.
- Boundary information matters: Explicitly learning boundary features, even when dropping them, significantly impacts segmentation accuracy, especially for thin and small objects like cracks and seam breaks.
- Contextual attention enhances understanding: The CAA module in the context stream proves effective in capturing long-range dependencies, leading to substantial improvements, particularly for classes with diverse patterns like "Track."
- Addressing class imbalance is critical: The proposed weighting mechanism in the loss function is vital for preventing the model from ignoring minority classes, which is a common issue in real-world datasets.
Key Results
The proposed TB-Net consistently outperforms several state-of-the-art semantic segmentation methods, including DeepLabv3, DeepLabv3+, DenseASPP, and BiSeNet, on their newly introduced dataset.
- Quantitative: TB-Net achieved a mean Class Pixel Accuracy (CPA) of 54.28% and a mean Intersection-over-Union (IoU) of 38.07% on the test set. This represents significant improvements over the best-performing baseline (BiSeNet) which achieved 40.19% CPA and 23.25% IoU. Notably, TB-Net showed substantial gains in challenging classes like "Crack" and "Seambroken."
- Qualitative: Visual results show that TB-Net produces more accurate and complete segmentations, especially for small and thin diseases where other methods struggle. The boundary predictions from TB-Net also accurately outline the segmented objects, contributing to sharper segmentation results.
Limitations and Future Work
- Limitations: The authors acknowledge that the dataset is relatively small, and performance on minority classes can still be improved. While the model performs well, the complexity of the three-stream architecture might lead to higher computational costs compared to simpler models.
- Future Work: The authors suggest exploring more advanced attention mechanisms and further investigating methods to improve performance on minority classes. Expanding the dataset with more diverse pavement conditions and disease types is also a natural next step.
Summary updated