CARAFE++: Unified Content-Aware ReAssembly of FEatures

Jiaqi Wang, Kai Chen, Rui Xu, Ziwei Liu, Chen Change Loy, Dahua Lin

TL;DR

This paper introduces CARAFE++, a novel, unified operator for feature downsampling and upsampling in deep neural networks. By adaptively generating kernels based on local content, CARAFE++ significantly improves performance across various dense prediction tasks like object detection and semantic segmentation with minimal computational overhead.

What Problem Does This Paper Address?

Feature reassembly, which includes downsampling and upsampling, is a fundamental operation in modern convolutional neural networks, particularly for tasks requiring dense predictions (e.g., object detection, segmentation, inpainting). Traditional methods like pooling and interpolation rely on fixed, rule-based operations that often fail to capture rich semantic information or adapt to the specific content of the input. While convolutional and deconvolutional layers offer learnable kernels, they typically apply a single kernel across the entire image, leading to limitations in handling local variations and incurring significant computational costs, especially for larger receptive fields. This paper addresses the need for a feature reassembly operator that can effectively aggregate contextual information from a large receptive field, adapt to instance-specific content on-the-fly, and remain computationally efficient.

Methods

The proposed solution is CARAFE++ (Content-Aware ReAssembly of FEatures++), a universal and lightweight operator designed for both downsampling and upsampling. CARAFE++ operates in two main steps:

  1. Kernel Prediction Module: This module generates adaptive, content-aware reassembly kernels for each target location. It comprises three sub-modules:
  2. Channel Compressor: A 1x1 convolution reduces the input feature map's channel dimension to reduce computational cost.
  3. Content Encoder: A convolution layer takes the compressed features and encodes local content to predict reassembly kernels. The kernel size and stride are adjusted for downsampling and upsampling.
  4. Kernel Normalizer: A softmax activation is applied to the predicted kernels to ensure they sum to 1, enabling a weighted combination of features.
  5. Content-aware Reassembly Module: This module uses the predicted kernels to reassemble features from a predefined local region around the source location. It performs a weighted sum of neighboring features, where the weights are determined by the generated kernels. This allows CARAFE++ to selectively aggregate information based on its semantic relevance.

CARAFE++ is designed to be seamlessly integrated into existing network architectures, replacing standard pooling or interpolation layers.

Datasets

The paper evaluates CARAFE++ on several standard benchmarks:

  • MS COCO 2017: For object detection and instance segmentation.
  • ADE20k: For semantic segmentation.
  • Places: For image inpainting.
  • ImageNet-1k: For evaluating CARAFE++ as a backbone component in image classification.

Key Insights and Ideas

  • Content-Awareness is Crucial: The core insight is that feature reassembly should not be a fixed, generic operation. By making the reassembly kernels adaptive to the local content, CARAFE++ can better capture semantic nuances and improve performance on dense prediction tasks.
  • Large Receptive Field Aggregation: Unlike traditional methods that focus on local neighborhoods, CARAFE++ aggregates information from a larger receptive field, leading to richer feature representations.
  • Efficiency Through Adaptive Kernels: While adaptive kernels might sound computationally expensive, CARAFE++ achieves this efficiently by generating small, spatially adaptive kernels on-the-fly, avoiding the heavy computation of large, fixed kernels or full deconvolution layers.
  • Universality: The operator's design allows it to be used for both downsampling and upsampling, making it a versatile building block for various network architectures and tasks.
  • "Aha Moment" in Visualization: Figure 9 provides a compelling qualitative insight, showing how CARAFE++ reassembles features from semantically similar regions, demonstrating its ability to focus on relevant information and ignore irrelevant background.

Key Results

CARAFE++ consistently improves performance across various tasks with negligible computational overhead:

  • Object Detection: Boosts Faster R-CNN by 2.5% APbox and Mask R-CNN by 2.1% APmask on MS COCO.
  • Semantic Segmentation: Improves UperNet by 1.94% mIoU on ADE20k.
  • Image Inpainting: Enhances Global&Local by 1.35 dB PSNR on Places.
  • Image Classification: Achieves stable accuracy improvements when used as a backbone component.
  • Efficiency: The method introduces minimal additional computational cost, often with faster inference speeds compared to some alternatives. For example, it achieves better results than methods like Pixel Shuffle or Spatial Attention with lower latency.

Limitations and Future Work

  • Integer Factor Scaling: The current version of CARAFE++ supports feature reassembly by integer factors only.
  • Future Work: The authors suggest extending CARAFE++ to support arbitrary scale factors for reassembly, which would further broaden its applicability. They also highlight its potential as a strong building block for future deep network architectures.

Summary updated

Abstract

Feature reassembly, i.e. feature downsampling and upsampling, is a key operation in a number of modern convolutional network architectures, e.g., residual networks and feature pyramids. Its design is critical for dense prediction tasks such as object detection and semantic/instance segmentation. In this work, we propose unified Content-Aware ReAssembly of FEatures (CARAFE++), a universal, lightweight, and highly effective operator to fulfill this goal. CARAFE++ has several appealing properties: (1) Unlike conventional methods such as pooling and interpolation that only exploit sub-pixel neighborhood, CARAFE++ aggregates contextual information within a large receptive field. (2) Instead of using a fixed kernel for all samples (e.g. convolution and deconvolution), CARAFE++ generates adaptive kernels on-the-fly to enable instance-specific content-aware handling. (3) CARAFE++ introduces little computational overhead and can be readily integrated into modern network architectures. We conduct comprehensive evaluations on standard benchmarks in object detection, instance/semantic segmentation, and image inpainting. CARAFE++ shows consistent and substantial gains on mainstream methods across all the tasks with negligible computational overhead. It shows great potential to serve as a strong building block for modern deep networks.
BibTeX
@article{wang2022carafe,
  author = {Jiaqi Wang and Kai Chen and Rui Xu and Ziwei Liu and Chen Change Loy and Dahua Lin},
  title = {{CARAFE++: Unified Content-Aware ReAssembly of FEatures}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2022},
  url = {https://openreview.net/forum?id=TvsCBpbG9Vn},
}
Figure