TL;DR
AGPF (Asymmetric Global Pyramid Fusion) is a novel "neck" for computer vision models that replaces fixed, hand-designed feature fusion paths with a dynamic, attention-based selection mechanism. By allowing every output location to retrieve context from a shared multi-scale memory, AGPF achieves consistent performance gains across dozens of detection and segmentation tasks, proving that "where" a model looks for context is just as important as "what" it sees.
What Problem Does This Paper Address?
In modern computer vision, Feature Pyramid Networks (FPNs) are the standard way to handle objects at different scales. They create a "pyramid" of features, from high-resolution/fine-detail to low-resolution/high-semantic-context. Traditionally, we fuse these levels using fixed paths—usually a top-down pathway where coarse information is passed down to finer levels.
The authors of this paper ask a fundamental question: Does a specific pixel on a fine-resolution grid actually want information from the level immediately above it?
By analyzing attention patterns, they discovered that locations on the finest grids actually allocate about 75% of their attention to features two levels coarser, bypassing the adjacent level entirely. Furthermore, as an object grows in size, the model's preference shifts even further toward coarser features. Fixed fusion paths can't adapt to these dynamic needs, creating a bottleneck in how multi-scale information is utilized.
Methods
The authors propose Asymmetric Global Pyramid Fusion (AGPF). Instead of a series of convolutional fusion steps, AGPF treats the feature pyramid as a retrieval problem:
- Multiscale Memory: Features from all pyramid levels are projected into a shared memory. To keep the computation efficient, the authors use Asymmetric Memory Pooling: they spatially pool the high-resolution (token-heavy) levels while keeping the coarser levels relatively dense.
- Global Attention: Each location on the output grid acts as a "query" that attends to this shared memory. This allows a pixel to "pick and choose" exactly which scales and spatial regions provide the most useful context for its specific task.
- FA-RoPE (Feature-Aligned Rotary Position Embedding): Because different pyramid levels have different "strides" (pixel densities), a standard positional encoding won't work. The authors introduced FA-RoPE to map all tokens into a unified image-space coordinate system, ensuring the attention mechanism understands spatial relationships across different resolutions.
- Stackable Blocks: AGPF is designed as a modular block that can be stacked (AGPF-S, -B, -L) to replace standard FPNs without changing the rest of the detector's architecture.
Datasets
The researchers evaluated AGPF across several industry-standard benchmarks:
- COCO val2017: Used for object detection, instance segmentation, and panoptic segmentation.
- Cityscapes: Used for semantic segmentation in urban environments.
- ADE20K: Used for general-purpose semantic segmentation.
Key Insights and Ideas
- The "Non-Adjacent" Preference: The most surprising finding was that fine-grained queries (P2) prefer context from P4 (two levels up) over P3. This suggests that traditional step-by-step top-down fusion might be diluting the most useful semantic signals.
- Scale-Dependent Routing: The model dynamically changes its "source selection" based on the object it is looking at. Even if the output resolution is fixed, the model shifts its attention toward coarser features as the target object gets larger.
- Spatial Dependency: AGPF exhibits a much broader spatial "receptive field" than FPN (38.3% vs 2.3% median radius) while still maintaining sharp local sensitivity. It effectively sees the "big picture" without losing the "fine details."
Key Results
AGPF demonstrated nearly universal improvements over the baseline FPN:
- Universal Gains: It improved performance in all 19 object detection and all 7 instance segmentation configurations tested on COCO.
- Detection/Segmentation: For a Cascade Mask R-CNN, a single AGPF block boosted Box AP by 1.7 points and Mask AP by 2.3 points.
- State-of-the-Art Integration: When combined with advanced heads like HTC++, AGPF-L achieved 55.2 Box AP and 48.0 Mask AP on COCO, outperforming many modern end-to-end transformer models like DINO and Co-DETR in efficiency-adjusted comparisons.
- Cross-Task Transfer: The gains were not limited to detection; they transferred consistently to semantic and panoptic segmentation tasks on Cityscapes and ADE20K.
Limitations and Future Work
- Small Object Masks: The authors noted that for very small objects, the mask AP sometimes lagged slightly behind FPN, suggesting that a single fusion block might need more refinement to capture ultra-fine spatial details.
- Computational Overhead: While the asymmetric pooling makes it practical, global attention is still more computationally expensive than simple convolutions. The "AGPF-T" (Tiny) variant was introduced to address memory-constrained scenarios.
- End-to-End Integration: Currently, AGPF acts as a "neck" for traditional detection heads. Future work could explore how this dynamic source selection integrates directly into the query-based decoding of DETR-style architectures.
Summary updated