Cityscapes-Panoptic-Parts and PASCAL-Panoptic-Parts datasets for Scene Understanding

Panagiotis Meletis, Xiaoxiao Wen, Chenyang Lu, Daan de Geus, Gijs DubbelmanarXiv 2020 · arXiv

This technical report introduces two new benchmarks for the computer vision community: Cityscapes-Panoptic-Parts and PASCAL-Panoptic-Parts. By combining holistic scene understanding (panoptic segmentation) with fine-grained object analysis (part segmentation), these datasets provide a consistent, multi-level hierarchy of labels that allows researchers to train and test algorithms across varying levels of granularity.

TL;DR

The authors present two novel datasets that bridge the gap between holistic scene understanding and detailed object part analysis by providing consistent semantic, instance, and part-level annotations. This unified framework enables researchers to tackle complex tasks like part-level panoptic segmentation without the conflicts inherent in using multiple disparate datasets.

What Problem Does This Paper Address?

In computer vision, scene understanding is often fragmented into distinct tasks:

  1. Semantic Segmentation: Labeling every pixel with a class (e.g., "road," "car").
  2. Instance Segmentation: Distinguishing between individual objects of the same class (e.g., "Car A" vs. "Car B").
  3. Panoptic Segmentation: A hybrid of the two, separating "stuff" (amorphous regions like sky or grass) from "things" (countable objects like cars or people).
  4. Part Segmentation: Identifying components within an object (e.g., "wheels" on a car, "arms" on a person).

Historically, researchers had to use different datasets for these tasks, leading to inconsistencies. A pixel might be labeled as a "person" in one dataset but lack the "arm" or "leg" labels found in another. This paper addresses the need for a unified hierarchy where every pixel is labeled at three levels—semantic, instance, and part—ensuring a conflict-free ground truth for multi-task learning.

Methods

The authors developed a three-level label hierarchy to provide a structured approach to scene understanding:

Figure 1: The label hierarchy according to which we provide annotations for and the related image understanding tasks that can be used.

Annotation and Merging

  • Cityscapes-Panoptic-Parts: The authors manually annotated 23 part-level classes for five "thing" classes (person, rider, car, truck, bus) within the existing Cityscapes dataset. They used a team of 11 annotators and three experts to ensure high-quality, consistent definitions.
  • PASCAL-Panoptic-Parts: Instead of new manual labels, the authors merged two existing datasets: PASCAL-Context (which has rich semantic labels) and PASCAL-Parts (which has instance and part labels). They developed a "best-effort" merging strategy to resolve inter-dataset conflicts (overlapping labels) and intra-dataset conflicts (overlapping parts, like a "nose" being part of a "head").

Compact Encoding

To store this complex hierarchy efficiently, the authors introduced a 7-digit integer ID for every pixel:

  • Digits 1-2: Semantic ID (0-99)
  • Digits 3-5: Instance ID (0-999)
  • Digits 6-7: Part ID (0-99)

This format allows a single image-like file to contain all three levels of information, making it compatible with standard deep learning pipelines.

Datasets

The report introduces two primary benchmarks:

  1. Cityscapes-Panoptic-Parts: Extends the urban street scene dataset. It includes 2,975 training and 500 validation images, featuring 23 part classes (e.g., torso, head, arms, legs for humans; windows, wheels, lights, chassis for vehicles).
  2. PASCAL-Panoptic-Parts: A compound dataset for generic scenes. It includes 10,103 images (train + val) with 80 "stuff" classes, 20 "things" classes, and a massive 193 part classes.

Key Insights and Ideas

  • Hierarchical Consistency: By providing all labels on the same set of images, the authors eliminate the "label noise" that occurs when trying to combine different datasets.
  • The "Stuff" vs. "Things" Distinction: The authors maintain the panoptic convention where only "things" (countable objects) typically have instance and part IDs, though they acknowledge that "stuff" (like a building having windows) could eventually have parts too.
  • Extreme Class Imbalance: The part-level data reveals a massive "long-tail" distribution. For example, a vehicle's "chassis" occupies far more pixels than its "license plate." This presents a significant challenge for future algorithm design.
Absolute number of pixels per semantic and part class, highlighting the extreme imbalance.

Key Results

The authors successfully generated dense, high-quality annotations for thousands of images.

  • In Cityscapes-Panoptic-Parts, they achieved an average of 15.6 instances per image, nearly all of which are now annotated with parts.
  • In PASCAL-Panoptic-Parts, they managed to condense 459 PASCAL-Context classes down to a more manageable 100 informative classes while preserving 193 part categories.

Qualitatively, the datasets provide much richer detail than their predecessors, as seen in the visual examples provided in the report:

Figure 2: Examples from Cityscapes-Panoptic-Parts images and labels. Top two rows: training split. Bottom two rows: validation split.

Limitations and Future Work

  • Stuff Parts: Currently, part-level annotations are only provided for "things" classes. The authors note that "stuff" classes (like clouds or windows in a building) also have parts, which they leave for future research.
  • Public Test Sets: Because the original Cityscapes and PASCAL test sets are not public, the authors could only provide annotations for the training and validation splits.
  • Selection Criteria: For PASCAL, the authors used a specific metric (average occupied area × number of images) to select the top 100 classes. They suggest that future work could explore different measures to include even more semantically important classes.

Summary updated

Abstract

In this technical report, we present two novel datasets for image scene understanding. Both datasets have annotations compatible with panoptic segmentation and additionally they have part-level labels for selected semantic classes. This report describes the format of the two datasets, the annotation protocols, the merging strategies, and presents the datasets statistics. The datasets labels together with code for processing and visualization will be published at https://github.com/tue-mps/panoptic_parts.
BibTeX
@misc{meletis2020cityscapes,
  author = {Panagiotis Meletis and Xiaoxiao Wen and Chenyang Lu and Daan de Geus and Gijs Dubbelman},
  title = {{Cityscapes-Panoptic-Parts and PASCAL-Panoptic-Parts datasets for Scene Understanding}},
  year = {2020},
  eprint = {2004.07944},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2004.07944},
}
Figure