Abstract
RailSem19 is the first large-scale public dataset specifically designed for semantic scene understanding in the rail domain. By providing 8,500 annotated sequences from the ego-perspective of trains and trams, it bridges a critical data gap in autonomous vehicle research, enabling better safety at rail-road intersections and moving us closer to autonomous rail operations.
Introduction
While autonomous driving research for road vehicles has exploded over the last decade, the rail domain has been largely neglected. Existing datasets like Cityscapes or Mapillary Vistas focus on the perspective of cars. When trains or tracks appear in these datasets, they are often underrepresented or labeled with low granularity (e.g., a single "rail" label that covers both the tracks and the ground between them).
This lack of data creates two major issues: safety at intersections, where autonomous cars struggle to interpret complex rail-road crossings, and rail automation, which requires high-quality, rail-specific data to identify switches, signals, and platforms from a train's unique viewpoint.
The RailSem19 Dataset
The primary contribution is the RailSem19 dataset, featuring 8,500 frames selected from 530 sequences across 38 countries. It contains over 110,000 annotations, including 1,000+ railway crossings and 1,200 tram scenes. The labels are designed to be compatible with the Cityscapes format, allowing researchers to easily mix road and rail data.
Methodology
The authors introduce a novel "label policy" and an automated workflow to generate high-quality training data without the prohibitive cost of full manual pixel-wise annotation.
- Functional Labeling: Instead of generic labels, they focus on rail-specific geometry. This includes distinguishing between different types of rails and identifying the state of railway switches (left, right, or unknown) based on the position of the "blades."
- Frame Selection via
egodist: To avoid redundancy in video sequences, they developed a metric calledegodistto automatically filter out static or redundant frames, ensuring a diverse dataset. - Automated Dense Labeling: The authors combined manual geometric annotations (like splines for rails) with "weakly supervised" background labels generated by a state-of-the-art road-scene segmentation network.
Key Insights and Results
The authors found that viewpoint matters significantly: models trained on "platform-view" images perform poorly when tested on "ego-view" images from the train's cabin. This highlights the necessity of ego-perspective datasets for autonomous navigation. Furthermore, identifying the state of the track (e.g., switch direction) is functionally more important for a train than simply knowing a switch exists.
Validation results were promising:
- Image Classification: Achieved over 90% accuracy for most rail-relevant classes using DenseNet161.
- Semantic Segmentation: Achieved a mean Intersection-over-Union (mIoU) of 57.6%. Notably, the model showed strong performance on "rail-track" (81.9% mIoU), proving that the automated dense labeling process produces effective training data.
Limitations and Future Work
Distinguishing between "switch-left" and "switch-right" remains a challenge, achieving only ~67% accuracy in some tests due to the visual differences being only a few pixels wide. Future work will focus on specialized sub-networks for these tasks and expanding the dataset to include more diverse global signaling systems and infrastructure types to ensure worldwide generalization.
Summary updated