Abstract
Automating rail surface inspection is critical for railway safety. However, existing datasets often lack the scale, resolution, and annotation precision required for robust real-world deployment. We present Rail-5k, a dataset comprising over 5,000 high-resolution images featuring 13 defect classes. Rail-5k is characterized by a long-tailed distribution and includes a large set of uncurated unlabeled data to challenge semi-supervised learning models. Our experiments demonstrate that while object detection suffices for discrete components, semantic segmentation is superior for texture-like defects such as cracks. Furthermore, we identify a performance degradation in standard semi-supervised approaches when faced with real-world image corruptions, establishing Rail-5k as a rigorous benchmark for robust computer vision.
Introduction
Automating rail surface inspection is a high-stakes task for infrastructure safety. While deep learning has revolutionized object detection, the "real world" of railways is far less predictable than standard benchmarks like COCO or ImageNet. Existing railway datasets are often too small, low-resolution, or lack the instance-level annotations needed to train robust models.
Real-world data presents two primary challenges: the Long-Tail Problem, where critical defects occur far less frequently than common ones, and Uncurated Data, where environmental factors like lighting, dirt, and motion blur corrupt image quality. Rail-5k was developed to bridge the gap between laboratory AI and these messy operational realities.
The Rail-5k Dataset
The Rail-5k dataset consists of two primary components:
- Labeled Set: 1,100 high-resolution (3648 × 2736) RGB images annotated by 10 railway experts. It identifies 13 defect classes, including spalling, corrugation, and various fastener issues.
- Unlabeled Set: 4,000 uncurated images containing real-world corruptions and domain shifts.
Unlike previous datasets such as RSDDs or Delft, Rail-5k provides dense, instance-level annotations and significantly higher resolution ($10^7$ pixels vs $10^4$). The dataset exhibits an extreme imbalance ratio of 40.98, accurately reflecting the long-tail nature of infrastructure decay.
Methodology
Researchers from Tongji University collected the data using specialized cameras mounted on inspection cars, positioned 200mm from the rail surface. To establish a performance baseline, two primary computer vision approaches were evaluated:
- Object Detection (YOLOv5-s): Used to identify 13 types of defects. Bounding boxes proved effective for distinct objects like screws but struggled with fine-grained textures.
- Semantic Segmentation (DeepLabv3): Specifically applied to the "crack" class. By treating cracks as a pixel-masking problem rather than a bounding-box problem, the model achieved significantly better results.
- Semi-Supervised Learning: A pseudo-labeling strategy was tested by training on labeled data, generating labels for the 4k unlabeled images, and retraining on the combined set.
Key Results and Insights
- Cracks as Textures: A critical finding was that standard object detection is ill-suited for rail cracks. Because cracks are thin and lack clear boundaries, performance was extremely low until switching to a segmentation-based approach, which achieved a 67.8% IoU.
- The Semi-Supervised Paradox: While adding unlabeled data typically improves performance in academic settings, the semi-supervised baseline for Rail-5k performed worse than the supervised one. The uncurated nature of the unlabeled images—full of noise and corruptions—misled the model, highlighting a major gap in current autonomous learning algorithms.
- Detection Performance: The YOLOv5-s model achieved a [email protected] of 63.29%. It performed exceptionally well on "Rail Surface" (98.9% AP) but struggled with "Indentation" (0.7% AP) due to the defect's subtle, fine-grained nature.
Conclusion
Rail-5k serves as a challenging benchmark for the next generation of robust inspection models. While currently limited to RGB imagery, future iterations are expected to include 3D-scan data for depth analysis and eddy current data to detect internal flaws. The dataset's primary contribution is exposing the "robustness gap" where current models fail to handle the uncurated, long-tailed data typical of real-world industrial environments.
Summary updated