Abstract
Panoptic segmentation requires a delicate balance between semantic consistency for background "stuff" and discriminative features for foreground "things." CFNet addresses this by learning task-specific correlation functions. By employing the Discrete Fourier Transform (DFT) to model spatial relationships, CFNet generates features tailored to the distinct requirements of semantic and instance segmentation. Our approach achieves state-of-the-art performance for one-stage methods on the MS COCO (45.1% PQ) and ADE20k (32.6% PQ) datasets.
Introduction
Panoptic segmentation is often described as the "grand challenge" of computer vision, requiring the simultaneous labeling of every pixel (semantic segmentation) and the delineation of individual object instances (instance segmentation). Most one-stage models utilize a shared backbone to extract features for both tasks. However, these tasks have fundamentally conflicting requirements: semantic segmentation seeks high intra-class consistency, while instance segmentation requires high instance-level discriminability.
Existing solutions, such as standard attention mechanisms, often struggle with the computational overhead of large correlation matrices or rely on static heuristics that fail in crowded scenes.
Methodology: The CFNet Framework
CFNet (Correlation Function Network) proposes a "box-free" one-stage pipeline that learns to predict correlations between pixels as continuous functions.
Architecture Overview
The system is built upon a Feature Pyramid Network (FPN) backbone. As shown in the architecture diagram, the features are branched into two specialized modules: the Instance Correlation Module (ICM) and the Semantic Correlation Module (SCM).
Correlation Functions via DFT
Instead of computing a dense, memory-intensive correlation matrix, CFNet treats the correlation between a point and its neighborhood as a sequence. By applying the Discrete Fourier Transform (DFT), the model approximates this sequence using a compact set of amplitudes and phases. This mathematical decomposition allows the network to capture complex spatial patterns with minimal parameters, inherently respecting spatial continuity.
Task-Specific Modules
- Semantic Correlation Module (SCM): SCM aggregates features across the scene to "smooth out" labels for background categories like grass or sky, ensuring that related pixels share information effectively.
- Instance Correlation Module (ICM): For object instances, the model calculates correlations relative to specific reference points. These maps provide the spatial cues necessary to distinguish between adjacent objects of the same class (e.g., two side-by-side giraffes).
Experimental Results
CFNet was evaluated on the MS COCO and ADE20k benchmarks, demonstrating superior performance among one-stage architectures.
Performance on MS COCO
On the MS COCO dataset, CFNet reached 45.1% PQ, surpassing previous leading models like Panoptic-FCN (44.3%). Notably, it maintained a competitive inference speed of approximately 10.5 FPS on a V100 GPU, proving its efficiency for real-world applications.
Performance on ADE20k
On the challenging ADE20k scene parsing dataset, CFNet achieved 32.6% PQ. This result is significant as it marks the first time a one-stage method has outperformed several established two-stage approaches on this benchmark.
Ablation Studies
Ablation experiments confirmed that the SCM and ICM provide synergistic benefits. Modeling correlations for both "stuff" and "things" leads to a more robust panoptic output than focusing on either task in isolation.
Conclusion
CFNet introduces a mathematically elegant solution to the feature conflict in panoptic segmentation. By treating pixel-to-pixel affinity as a signal to be decomposed via DFT, the model achieves high accuracy and efficiency. Future work may involve extending these correlation functions to video object segmentation or exploring 2D Fourier descriptors to capture even more intricate spatial relationships.
Summary updated