Traffic Scene Parsing through the TSP6K Dataset

Peng-Tao Jiang, Yuqi Yang, Yang Cao, Qibin Hou, Ming-Ming Cheng, Chunhua Shen

TL;DR

Researchers have introduced TSP6K, the largest dataset specifically designed for traffic monitoring scene parsing, featuring 6,000 high-resolution images with dense pixel-level annotations. To tackle the unique challenges of monitoring views—such as extreme scale variance and crowded scenes—they also proposed a Detail Refining Decoder (DRD) that uses region tokens to significantly improve segmentation accuracy for small objects.

What Problem Does This Paper Address?

Most computer vision research for smart cities relies on datasets like Cityscapes or KITTI, which are captured from the perspective of a car (driving-view). However, urban infrastructure relies heavily on high-hanging monitoring cameras (4.5–6 meters up).

This perspective shift creates a massive domain gap. Monitoring cameras see much more crowded scenes (often at intersections) and experience extreme scale variance—objects in the foreground are massive, while those in the distance are tiny. Models trained on driving datasets perform poorly in these monitoring scenarios because they aren't used to the "bird's-eye" angle or the sheer density of traffic participants.

Figure 1. Examples are randomly picked from the TSP6K dataset. Each image is associated with its corresponding semantic label and instance label. We have masked the vehicle plates for privacy protection.

Methods

The authors propose a new architecture featuring a Detail Refining Decoder (DRD) built upon a SegNeXt backbone. The core innovation is the Region Refining Module.

Standard self-attention mechanisms are computationally expensive on high-resolution images because every pixel "looks" at every other pixel. To solve this, the DRD introduces region tokens—a small set of learnable vectors.

  1. Cross-Attention: The model computes attention between these region tokens and the high-resolution features. This allows the tokens to "learn" representations for different semantic regions (e.g., one token might focus on the road, another on vehicles).
  2. Refinement: These learned region embeddings are then used to refine the pixel-level features.
  3. Efficiency: By using a few tokens instead of pixel-to-pixel attention, the model captures global context and fine details without the massive memory overhead.

Datasets

The primary contribution is the TSP6K dataset:

  • Scale: 6,000 images collected from hundreds of scenes across 10 Chinese provinces.
  • Annotations: High-quality pixel-level semantic labels (21 classes) and instance-level labels.
  • Diversity: Includes various weather conditions (sunny, fog, snow, rain) and times of day.
  • Comparison: The paper evaluates TSP6K against established driving datasets like Cityscapes, KITTI, BDD100K, and Mapillary Vistas, as well as synthetic datasets like SYNTHIA for domain adaptation tasks.
Class and scene information showing the distribution of annotated pixels and geographic spread.

Key Insights and Ideas

  • Density is the Difference: TSP6K is significantly more crowded than driving datasets. It averages 42 traffic participants per image, compared to just 18.8 in Cityscapes. Over 1,200 images in TSP6K contain more than 50 participants.
  • The Scale Challenge: Because of the high-hanging camera angle, the variance in instance sizes is much broader than in driving-view datasets. Small objects in the distance are a major failure point for existing models.
  • Domain Gap is Real: Unsupervised Domain Adaptation (UDA) experiments showed that even state-of-the-art models like HRDA, which achieve ~66% mIoU on Cityscapes, drop to ~45% when adapted to TSP6K, highlighting how different monitoring scenes truly are.
  • Region Tokens vs. Class Tokens: The authors found that using a small number of generic "region tokens" (e.g., 5) performed better and was more flexible than using one token per class, especially as the number of classes in a dataset grows.
Figure 3. Data analysis of the TSP6K dataset. (a) The distribution of the number of instances in each image. (b) The distribution of the instance sizes. (c) The number of instances for each category.

Key Results

The proposed DRD method sets a new state-of-the-art on the TSP6K dataset:

  • Scene Parsing: Achieved 75.8% mIoU and 58.4% iIoU (instance-level IoU) on the validation set.
  • Superiority: It outperformed the previous best model, SegNeXt, by 1.2% mIoU and 1.1% iIoU, while maintaining competitive computational efficiency (90.1 GFlops).
  • Small Object Recovery: Qualitative results show the DRD is much better at recovering sharp details for small objects, like distant arrows on the road or small pedestrians, compared to standard decoders.

Limitations and Future Work

  • Geographic Bias: The current dataset is primarily collected in China, meaning it lacks scenes from left-hand driving countries or significantly different urban architectures.
  • Modality: The dataset is limited to RGB images. Future work could incorporate multi-modal data like LiDAR or thermal imaging to improve robustness in nighttime or extreme weather.
  • Unlabeled Data: While 6,000 images are annotated, the authors collected a much larger pool of unlabeled data that could be leveraged for semi-supervised learning in the future.

Summary updated

Abstract

Traffic scene perception in computer vision is a critically important task to achieve intelligent cities. To date, most existing datasets focus on autonomous driving scenes. We observe that the models trained on those driving datasets often yield unsatisfactory results on traffic monitoring scenes. However, little effort has been put into improving the traffic monitoring scene understanding, mainly due to the lack of specific datasets. To fill this gap, we introduce a specialized traffic monitoring dataset, termed TSP6K, containing images from the traffic monitoring scenario, with high-quality pixel-level and instance-level annotations. The TSP6K dataset captures more crowded traffic scenes with several times more traffic participants than the existing driving scenes. We perform a detailed analysis of the dataset and comprehensively evaluate previous popular scene parsing methods, instance segmentation methods and unsupervised domain adaption methods. Furthermore, considering the vast difference in instance sizes, we propose a detail refining decoder for scene parsing, which recovers the details of different semantic regions in traffic scenes owing to the proposed TSP6K dataset. Experiments show its effectiveness in parsing the traffic monitoring scenes. Code and dataset are available at https://github.com/PengtaoJiang/TSP6K.
BibTeX
@inproceedings{jiang2023traffic,
  author = {Peng-Tao Jiang and Yuqi Yang and Yang Cao and Qibin Hou and Ming-Ming Cheng and Chunhua Shen},
  title = {{Traffic Scene Parsing through the TSP6K Dataset}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure