Abstract
Semantic segmentation is a critical component for autonomous driving, yet existing benchmarks often lack the temporal resolution necessary for real-time video applications. This paper introduces the Highway Driving dataset, the first high-frequency (30Hz) benchmark with dense semantic annotations. Alongside this dataset, we propose a "Priming-Approximating" baseline algorithm that leverages temporal correlation to achieve near-state-of-the-art accuracy while significantly reducing computational overhead.
Introduction
While semantic segmentation—assigning class labels to every pixel—has advanced rapidly, most research focuses on static image segmentation. Existing video datasets like Cityscapes provide high-quality labels but treat frames as independent images. Others, such as CamVid, offer video sequences but only annotate frames at 1Hz. For a vehicle at highway speeds, 1Hz is insufficient to capture fluid motion. This "temporal gap" hinders the development of models that use temporal priors to accelerate processing.
The Highway Driving Dataset
The Highway Driving dataset addresses these limitations by providing high-frequency, spatially dense annotations.
- Scale and Frequency: 20 video sequences (2 seconds each) at 30Hz, totaling 1,200 fully annotated frames.
- Density: 97.8% of pixels are annotated across 10 categories (Road, Lane, Sky, Fence, Construction, Traffic Sign, Car, Truck, Vegetation, and Unknown).
- Quality Control: An adversarial annotation process ensures temporal consistency, preventing label "jitter" between frames.
Proposed Method: Priming-Approximating Network
To exploit the high temporal correlation in 30Hz video, we propose a three-part architecture designed for efficiency:
- Priming Network: A deep Dilated Residual Network (DRN) processes the initial frame to provide a highly accurate "anchor" or prior.
- Approximating Network: Subsequent frames are processed by a lighter network using downsampled inputs to generate a rough segmentation.
- Ensemble Network: A shallow network fuses the accurate prior from the previous frame with the current rough approximation. This step is crucial for recovering fine details like lane markings, which are often lost during downsampling.
By running the heavy Priming network only occasionally, the system maintains high accuracy while drastically reducing the per-frame time budget.
Key Results
Evaluations on the Highway Driving dataset demonstrate the efficiency of this approach. Our model achieved a Mean IoU (mIoU) of 55.7 with a relative runtime of only 0.36 compared to the baseline DRN. The system allows for a flexible trade-off between speed and accuracy by adjusting the priming frequency. Furthermore, experiments on the CamVid dataset confirmed the model's generalizability, maintaining competitive accuracy (66.5% Class Avg.) while running nearly 3x faster than the priming network alone.
Conclusion and Future Work
The Highway Driving dataset provides a unique resource for studying temporal redundancy in semantic segmentation. While currently focused on highway scenes, future expansions could include diverse weather and urban environments. Future research will also explore integrating optical flow into the Ensemble Network to better track fast-moving objects and further refine fine details.
Summary updated