Highway Driving Dataset for Semantic Video Segmentation

Byungju Kim, Junho Yim, Junmo KimarXiv 2020 · arXiv

Abstract

Semantic segmentation is a critical component for autonomous driving, yet existing benchmarks often lack the temporal resolution necessary for real-time video applications. This paper introduces the Highway Driving dataset, the first high-frequency (30Hz) benchmark with dense semantic annotations. Alongside this dataset, we propose a "Priming-Approximating" baseline algorithm that leverages temporal correlation to achieve near-state-of-the-art accuracy while significantly reducing computational overhead.

Introduction

While semantic segmentation—assigning class labels to every pixel—has advanced rapidly, most research focuses on static image segmentation. Existing video datasets like Cityscapes provide high-quality labels but treat frames as independent images. Others, such as CamVid, offer video sequences but only annotate frames at 1Hz. For a vehicle at highway speeds, 1Hz is insufficient to capture fluid motion. This "temporal gap" hinders the development of models that use temporal priors to accelerate processing.

The Highway Driving Dataset

The Highway Driving dataset addresses these limitations by providing high-frequency, spatially dense annotations.

  • Scale and Frequency: 20 video sequences (2 seconds each) at 30Hz, totaling 1,200 fully annotated frames.
  • Density: 97.8% of pixels are annotated across 10 categories (Road, Lane, Sky, Fence, Construction, Traffic Sign, Car, Truck, Vegetation, and Unknown).
  • Quality Control: An adversarial annotation process ensures temporal consistency, preventing label "jitter" between frames.
Sample frames and dense semantic annotations from the Highway Driving dataset.

Proposed Method: Priming-Approximating Network

To exploit the high temporal correlation in 30Hz video, we propose a three-part architecture designed for efficiency:

  1. Priming Network: A deep Dilated Residual Network (DRN) processes the initial frame to provide a highly accurate "anchor" or prior.
  2. Approximating Network: Subsequent frames are processed by a lighter network using downsampled inputs to generate a rough segmentation.
  3. Ensemble Network: A shallow network fuses the accurate prior from the previous frame with the current rough approximation. This step is crucial for recovering fine details like lane markings, which are often lost during downsampling.
Proposed Priming-Approximating architecture for efficient video segmentation.

By running the heavy Priming network only occasionally, the system maintains high accuracy while drastically reducing the per-frame time budget.

Key Results

Evaluations on the Highway Driving dataset demonstrate the efficiency of this approach. Our model achieved a Mean IoU (mIoU) of 55.7 with a relative runtime of only 0.36 compared to the baseline DRN. The system allows for a flexible trade-off between speed and accuracy by adjusting the priming frequency. Furthermore, experiments on the CamVid dataset confirmed the model's generalizability, maintaining competitive accuracy (66.5% Class Avg.) while running nearly 3x faster than the priming network alone.

Conclusion and Future Work

The Highway Driving dataset provides a unique resource for studying temporal redundancy in semantic segmentation. While currently focused on highway scenes, future expansions could include diverse weather and urban environments. Future research will also explore integrating optical flow into the Ensemble Network to better track fast-moving objects and further refine fine details.

Summary updated

Abstract

Scene understanding is an essential technique in semantic segmentation. Although there exist several datasets that can be used for semantic segmentation, they are mainly focused on semantic image segmentation with large deep neural networks. Therefore, these networks are not useful for real time applications, especially in autonomous driving systems. In order to solve this problem, we make two contributions to semantic segmentation task. The first contribution is that we introduce the semantic video dataset, the Highway Driving dataset, which is a densely annotated benchmark for a semantic video segmentation task. The Highway Driving dataset consists of 20 video sequences having a 30Hz frame rate, and every frame is densely annotated. Secondly, we propose a baseline algorithm that utilizes a temporal correlation. Together with our attempt to analyze the temporal correlation, we expect the Highway Driving dataset to encourage research on semantic video segmentation.
BibTeX
@misc{kim2020highway,
  author = {Byungju Kim and Junho Yim and Junmo Kim},
  title = {{Highway Driving Dataset for Semantic Video Segmentation}},
  year = {2020},
  eprint = {2011.00674},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2011.00674},
}
Figure