This technical report introduces two new benchmarks for the computer vision community: Cityscapes-Panoptic-Parts and PASCAL-Panoptic-Parts. By combining holistic scene understanding (panoptic segmentation) with fine-grained object analysis (part segmentation), these datasets provide a consistent, multi-level hierarchy of labels that allows researchers to train and test algorithms across varying levels of granularity.
TL;DR
The authors present two novel datasets that bridge the gap between holistic scene understanding and detailed object part analysis by providing consistent semantic, instance, and part-level annotations. This unified framework enables researchers to tackle complex tasks like part-level panoptic segmentation without the conflicts inherent in using multiple disparate datasets.
What Problem Does This Paper Address?
In computer vision, scene understanding is often fragmented into distinct tasks:
- Semantic Segmentation: Labeling every pixel with a class (e.g., "road," "car").
- Instance Segmentation: Distinguishing between individual objects of the same class (e.g., "Car A" vs. "Car B").
- Panoptic Segmentation: A hybrid of the two, separating "stuff" (amorphous regions like sky or grass) from "things" (countable objects like cars or people).
- Part Segmentation: Identifying components within an object (e.g., "wheels" on a car, "arms" on a person).
Historically, researchers had to use different datasets for these tasks, leading to inconsistencies. A pixel might be labeled as a "person" in one dataset but lack the "arm" or "leg" labels found in another. This paper addresses the need for a unified hierarchy where every pixel is labeled at three levels—semantic, instance, and part—ensuring a conflict-free ground truth for multi-task learning.
Methods
The authors developed a three-level label hierarchy to provide a structured approach to scene understanding:
Annotation and Merging
- Cityscapes-Panoptic-Parts: The authors manually annotated 23 part-level classes for five "thing" classes (person, rider, car, truck, bus) within the existing Cityscapes dataset. They used a team of 11 annotators and three experts to ensure high-quality, consistent definitions.
- PASCAL-Panoptic-Parts: Instead of new manual labels, the authors merged two existing datasets: PASCAL-Context (which has rich semantic labels) and PASCAL-Parts (which has instance and part labels). They developed a "best-effort" merging strategy to resolve inter-dataset conflicts (overlapping labels) and intra-dataset conflicts (overlapping parts, like a "nose" being part of a "head").
Compact Encoding
To store this complex hierarchy efficiently, the authors introduced a 7-digit integer ID for every pixel:
- Digits 1-2: Semantic ID (0-99)
- Digits 3-5: Instance ID (0-999)
- Digits 6-7: Part ID (0-99)
This format allows a single image-like file to contain all three levels of information, making it compatible with standard deep learning pipelines.
Datasets
The report introduces two primary benchmarks:
- Cityscapes-Panoptic-Parts: Extends the urban street scene dataset. It includes 2,975 training and 500 validation images, featuring 23 part classes (e.g., torso, head, arms, legs for humans; windows, wheels, lights, chassis for vehicles).
- PASCAL-Panoptic-Parts: A compound dataset for generic scenes. It includes 10,103 images (train + val) with 80 "stuff" classes, 20 "things" classes, and a massive 193 part classes.
Key Insights and Ideas
- Hierarchical Consistency: By providing all labels on the same set of images, the authors eliminate the "label noise" that occurs when trying to combine different datasets.
- The "Stuff" vs. "Things" Distinction: The authors maintain the panoptic convention where only "things" (countable objects) typically have instance and part IDs, though they acknowledge that "stuff" (like a building having windows) could eventually have parts too.
- Extreme Class Imbalance: The part-level data reveals a massive "long-tail" distribution. For example, a vehicle's "chassis" occupies far more pixels than its "license plate." This presents a significant challenge for future algorithm design.
Key Results
The authors successfully generated dense, high-quality annotations for thousands of images.
- In Cityscapes-Panoptic-Parts, they achieved an average of 15.6 instances per image, nearly all of which are now annotated with parts.
- In PASCAL-Panoptic-Parts, they managed to condense 459 PASCAL-Context classes down to a more manageable 100 informative classes while preserving 193 part categories.
Qualitatively, the datasets provide much richer detail than their predecessors, as seen in the visual examples provided in the report:
Limitations and Future Work
- Stuff Parts: Currently, part-level annotations are only provided for "things" classes. The authors note that "stuff" classes (like clouds or windows in a building) also have parts, which they leave for future research.
- Public Test Sets: Because the original Cityscapes and PASCAL test sets are not public, the authors could only provide annotations for the training and validation splits.
- Selection Criteria: For PASCAL, the authors used a specific metric (average occupied area × number of images) to select the top 100 classes. They suggest that future work could explore different measures to include even more semantically important classes.
Summary updated