Panoptic SwiftNet: Pyramidal Fusion for Real-time Panoptic Segmentation

Josip Saric, Marin Orsic, Sinisa Segvic

Abstract

Panoptic segmentation, the simultaneous task of semantic and instance segmentation, remains computationally demanding for real-time applications. We present Panoptic SwiftNet, a novel architecture that achieves high-speed panoptic segmentation on high-resolution images. By employing a lightweight backbone with a pyramidal fusion scheme and a boundary-aware loss function, our model delivers state-of-the-art results on aerial and urban datasets. Notably, Panoptic SwiftNet processes 1MPx images at over 100 FPS on modern GPUs using FP16 precision, making it highly suitable for autonomous driving and large-scale remote sensing.

Introduction

Panoptic segmentation is often considered the "holy grail" of computer vision recognition, requiring models to label every pixel (semantic segmentation) while simultaneously identifying and separating individual object instances (instance segmentation). Current state-of-the-art models typically rely on heavy backbones to achieve a sufficient receptive field, which limits their utility in real-time scenarios like autonomous driving or large-scale aerial image processing. Furthermore, bottom-up instance segmentation methods often struggle with precision at object boundaries. Panoptic SwiftNet addresses these challenges by prioritizing efficiency and multi-scale context.

Methods

The core of Panoptic SwiftNet is designed for high-speed inference without compromising accuracy through several key innovations.

Pyramidal Fusion

Instead of a single high-resolution pass, the model utilizes a three-level image pyramid (original, 1/2, and 1/4 scale). Features are extracted from all three scales and fused via a cross-scale upsampling path. This pyramidal fusion allows the network to capture global context from low-resolution inputs while maintaining spatial precision from high-resolution features, effectively replacing the need for deeper, slower backbones.

Architecture of Panoptic SwiftNet showing the pyramidal fusion and shared decoder.

Shared Decoder and BAOL

The architecture employs a shared upsampling path for three distinct heads: semantic segmentation, instance center heatmaps, and offset vectors. To improve instance separation, the authors introduced Boundary-Aware Offset Learning (BAOL). This weighted L1 loss prioritizes training on pixels near instance edges—where offset vectors typically flip direction—solving common consistency issues where touching objects might otherwise be merged into a single instance.

Results

Panoptic SwiftNet was evaluated across four diverse benchmarks: BSB-Aerial, Cityscapes, Mapillary Vistas, and COCO.

On the BSB-Aerial dataset, the model achieved a Panoptic Quality (PQ) of 68.1, outperforming the previous state-of-the-art (Panoptic FPN) by 3.1 points while maintaining significantly higher inference speeds. On Cityscapes, it reached 55.9 PQ, proving competitive with much heavier models like UPSNet while maintaining a 2x to 4x speed advantage.

Quantitative results on the BSB-Aerial dataset comparing Panoptic SwiftNet to baselines.

Ablation studies confirmed that the pyramidal fusion scheme is critical, contributing over 4 points to the PQ on Cityscapes compared to a single-scale baseline. When optimized with TensorRT and custom CUDA kernels for post-processing, the model achieves over 100 FPS on an RTX 3090 at 1MPx resolution.

Conclusion

Panoptic SwiftNet demonstrates that real-time panoptic segmentation is achievable through efficient multi-scale feature fusion and targeted loss functions. While the model shows some limitations with very small, distant objects, its speed and accuracy on high-resolution data set a new standard for practical panoptic recognition. Future work will focus on improving semantic accuracy and further streamlining the post-processing pipeline for end-to-end efficiency.

Summary updated

BibTeX
@article{saric2023panoptic,
  author = {Josip Saric and Marin Orsic and Sinisa Segvic},
  title = {{Panoptic SwiftNet: Pyramidal Fusion for Real-time Panoptic Segmentation}},
  journal = {Remote Sensing},
  year = {2023},
  url = {https://openreview.net/forum?id=BKQflr8W2l},
}
Figure