Mask-Pyramid Network: A Novel Panoptic Segmentation Method

Pengfei Xian, Lai-Man Po, Jingjing Xiong, Yuzhi Zhao, Wing Yin Yu, Kwok-Wai Cheung

TL;DR

This paper introduces the Mask-Pyramid Network (Mask-PNet), a novel approach to panoptic segmentation that significantly reduces computational cost by intelligently generating fewer object proposals. By leveraging a pyramid-like strategy and the concept of "coverage," Mask-PNet achieves competitive accuracy while being more efficient than existing Mask R-CNN based methods.

What Problem Does This Paper Address?

Panoptic segmentation aims to provide a complete scene understanding by combining semantic segmentation (labeling every pixel with a class like "road" or "sky") and instance segmentation (identifying and segmenting individual objects like "car" or "person"). Traditional methods, often built upon Mask R-CNN, struggle with computational inefficiency due to generating a vast number of object proposals that are later discarded. Furthermore, effectively merging the outputs of semantic and instance segmentation into a unified representation remains a challenge. This paper tackles these issues by proposing a more efficient proposal generation mechanism and a natural fusion strategy.

Methods

The core innovation of Mask-PNet lies in its Mask Pyramid (MP) branch and its proposal generation strategy. Unlike Mask R-CNN, which generates proposals at multiple feature levels and then filters them, Mask-PNet employs a pyramid-based approach. It starts by detecting larger objects at coarser feature levels and uses their predicted masks to define "occupied" areas. Subsequent proposal generation is then restricted to the "unoccupied" areas, drastically reducing the number of proposals. This process is repeated level by level, from larger to smaller objects, mimicking a pyramid structure.

The method also introduces a novel way to fuse semantic and instance segmentation. Instead of separate post-processing steps, Mask-PNet generates "mask logits" for both "stuff" (semantic) and "thing" (instance) categories. These logits are then concatenated into a unified tensor, allowing for a natural fusion using SoftMax. This unified representation facilitates end-to-end training and gradient flow between the semantic and instance branches.

Datasets

The Mask-PNet was trained and evaluated on two standard panoptic segmentation benchmarks:

  • Cityscapes: Focuses on urban street scenes and is known for its complexity due to dynamic objects and cluttered environments.
  • COCO: A large-scale dataset commonly used for object detection and segmentation tasks.

Key Insights and Ideas

  • Intelligent Proposal Reduction: The most significant insight is that by leveraging the spatial coverage of already detected objects, we can avoid generating redundant proposals, leading to substantial computational savings.
  • Pyramid-based Generation: Processing objects from large to small in a pyramid fashion is an effective strategy for managing complexity and ensuring that larger, more dominant objects are handled first.
  • Unified Logit Fusion: Merging semantic and instance segmentation at the logit level, rather than as final masks, simplifies the fusion process and enables better end-to-end learning.
  • "Stuff" as Background Reference: The semantic segmentation branch's "stuff" predictions serve as a crucial background reference, guiding the instance segmentation process and improving the accuracy of "stuff" regions in the final output.

Key Results

  • Accuracy: Mask-PNet achieves comparable accuracy to state-of-the-art methods on both Cityscapes and COCO datasets. On Cityscapes, it outperforms UPSNet by 1.98% in Panoptic Quality (PQ).
  • Efficiency: The proposed method significantly reduces computational resources. Compared to Mask R-CNN, Mask-PNet uses 14% fewer FLOPs and 17.6% less GPU memory. The number of generated proposals is drastically reduced (e.g., 951 for Mask-PNet vs. 65,472 for Mask R-CNN on a specific test).
  • Qualitative Improvements: Visualizations show that Mask-PNet produces sharper and more accurate segmentation edges compared to Mask R-CNN.

Limitations and Future Work

  • Computational Cost vs. Speed: While Mask-PNet is more computationally efficient in terms of FLOPs and memory, it might spend more time in prediction rounds, potentially making it slower on smaller devices compared to some highly optimized methods like MaskConver.
  • Future Directions: The authors plan to explore parallel algorithms for object detection applications and extend the Mask-PNet architecture to video segmentation and understanding. They also aim to improve its performance on smaller devices.

Summary updated

Abstract

In this paper, we introduce a novel panoptic segmentation method called the Mask-Pyramid Network. Existing Mask RCNN-based methods first generate a large number of box proposals and then filter them at each feature level, which requires a lot of computational resources, while most of the box proposals are suppressed and discarded in the Non-Maximum Suppression process. Additionally, for panoptic segmentation, it is a problem to properly fuse the semantic segmentation results with the Mask RCNN-produced instance segmentation results. To address these issues, we propose a new mask pyramid mechanism to distinguish objects and generate much fewer proposals by referring to existing segmented masks, so as to reduce computing resource consumption. The Mask-Pyramid Network generates object proposals and predicts masks from larger to smaller sizes. It records the pixel area occupied by the larger object masks, and then only generates proposals on the unoccupied areas. Each object mask is represented as a H × W × 1 logit, which fits well in format with the semantic segmentation logits. By applying SoftMax to the concatenated semantic and instance segmentation logits, it is easy and natural to fuse both segmentation results. We empirically demonstrate that the proposed Mask-Pyramid Network achieves comparable accuracy performance on the Cityscapes and COCO datasets. Furthermore, we demonstrate the computational efficiency of the proposed method and obtain competitive results.
BibTeX
@inproceedings{xian2024mask,
  author = {Pengfei Xian and Lai-Man Po and Jingjing Xiong and Yuzhi Zhao and Wing Yin Yu and Kwok-Wai Cheung},
  title = {{Mask-Pyramid Network: A Novel Panoptic Segmentation Method}},
  booktitle = {Sensors},
  year = {2024},
  url = {https://openreview.net/forum?id=6M2uKCfhhP},
}
Figure