CFNet: Learning Correlation Functions for One-Stage Panoptic Segmentation

Yifeng Chen, Wenqing Chu, Fangfang Wang, Ying Tai, Ran Yi, Zhenye Gan, Liang Yao, Chengjie Wang, Xi LiarXiv 2022 · arXiv

Abstract

Panoptic segmentation requires a delicate balance between semantic consistency for background "stuff" and discriminative features for foreground "things." CFNet addresses this by learning task-specific correlation functions. By employing the Discrete Fourier Transform (DFT) to model spatial relationships, CFNet generates features tailored to the distinct requirements of semantic and instance segmentation. Our approach achieves state-of-the-art performance for one-stage methods on the MS COCO (45.1% PQ) and ADE20k (32.6% PQ) datasets.

Introduction

Panoptic segmentation is often described as the "grand challenge" of computer vision, requiring the simultaneous labeling of every pixel (semantic segmentation) and the delineation of individual object instances (instance segmentation). Most one-stage models utilize a shared backbone to extract features for both tasks. However, these tasks have fundamentally conflicting requirements: semantic segmentation seeks high intra-class consistency, while instance segmentation requires high instance-level discriminability.

Comparison of instance and semantic correlation requirements.

Existing solutions, such as standard attention mechanisms, often struggle with the computational overhead of large correlation matrices or rely on static heuristics that fail in crowded scenes.

Methodology: The CFNet Framework

CFNet (Correlation Function Network) proposes a "box-free" one-stage pipeline that learns to predict correlations between pixels as continuous functions.

Architecture Overview

The system is built upon a Feature Pyramid Network (FPN) backbone. As shown in the architecture diagram, the features are branched into two specialized modules: the Instance Correlation Module (ICM) and the Semantic Correlation Module (SCM).

The CFNet architecture overview.

Correlation Functions via DFT

Instead of computing a dense, memory-intensive correlation matrix, CFNet treats the correlation between a point and its neighborhood as a sequence. By applying the Discrete Fourier Transform (DFT), the model approximates this sequence using a compact set of amplitudes and phases. This mathematical decomposition allows the network to capture complex spatial patterns with minimal parameters, inherently respecting spatial continuity.

Task-Specific Modules

  • Semantic Correlation Module (SCM): SCM aggregates features across the scene to "smooth out" labels for background categories like grass or sky, ensuring that related pixels share information effectively.
  • Instance Correlation Module (ICM): For object instances, the model calculates correlations relative to specific reference points. These maps provide the spatial cues necessary to distinguish between adjacent objects of the same class (e.g., two side-by-side giraffes).

Experimental Results

CFNet was evaluated on the MS COCO and ADE20k benchmarks, demonstrating superior performance among one-stage architectures.

Performance on MS COCO

On the MS COCO dataset, CFNet reached 45.1% PQ, surpassing previous leading models like Panoptic-FCN (44.3%). Notably, it maintained a competitive inference speed of approximately 10.5 FPS on a V100 GPU, proving its efficiency for real-world applications.

Performance on ADE20k

On the challenging ADE20k scene parsing dataset, CFNet achieved 32.6% PQ. This result is significant as it marks the first time a one-stage method has outperformed several established two-stage approaches on this benchmark.

Ablation Studies

Ablation experiments confirmed that the SCM and ICM provide synergistic benefits. Modeling correlations for both "stuff" and "things" leads to a more robust panoptic output than focusing on either task in isolation.

Conclusion

CFNet introduces a mathematically elegant solution to the feature conflict in panoptic segmentation. By treating pixel-to-pixel affinity as a signal to be decomposed via DFT, the model achieves high accuracy and efficiency. Future work may involve extending these correlation functions to video object segmentation or exploring 2D Fourier descriptors to capture even more intricate spatial relationships.

Summary updated

Abstract

Recently, there is growing attention on one-stage panoptic segmentation methods which aim to segment instances and stuff jointly within a fully convolutional pipeline efficiently. However, most of the existing works directly feed the backbone features to various segmentation heads ignoring the demands for semantic and instance segmentation are different: The former needs semantic-level discriminative features, while the latter requires features to be distinguishable across instances. To alleviate this, we propose to first predict semantic-level and instance-level correlations among different locations that are utilized to enhance the backbone features, and then feed the improved discriminative features into the corresponding segmentation heads, respectively. Specifically, we organize the correlations between a given location and all locations as a continuous sequence and predict it as a whole. Considering that such a sequence can be extremely complicated, we adopt Discrete Fourier Transform (DFT), a tool that can approximate an arbitrary sequence parameterized by amplitudes and phrases. For different tasks, we generate these parameters from the backbone features in a fully convolutional way which is optimized implicitly by corresponding tasks. As a result, these accurate and consistent correlations contribute to producing plausible discriminative features which meet the requirements of the complicated panoptic segmentation task. To verify the effectiveness of our methods, we conduct experiments on several challenging panoptic segmentation datasets and achieve state-of-the-art performance on MS COCO with $45.1$\% PQ and ADE20k with $32.6$\% PQ.
BibTeX
@misc{chen2022cfnet,
  author = {Yifeng Chen and Wenqing Chu and Fangfang Wang and Ying Tai and Ran Yi and Zhenye Gan and Liang Yao and Chengjie Wang and Xi Li},
  title = {{CFNet: Learning Correlation Functions for One-Stage Panoptic Segmentation}},
  year = {2022},
  eprint = {2201.04796},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2201.04796},
}
Figure