CARAFE++: Unified Content-Aware ReAssembly of FEatures

Jiaqi Wang, Kai Chen, Rui Xu, Ziwei Liu, Chen Change Loy, Dahua Lin

Abstract

Feature reassembly, i.e. feature downsampling and upsampling, is a key operation in a number of modern convolutional network architectures, e.g., residual networks and feature pyramids. Its design is critical for dense prediction tasks such as object detection and semantic/instance segmentation. In this work, we propose unified Content-Aware ReAssembly of FEatures (CARAFE++), a universal, lightweight and highly effective operator to fulfill this goal. CARAFE++ has several appealing properties: (1) Unlike conventional methods such as pooling and interpolation that only exploit sub-pixel neighborhood, CARAFE++ aggregates contextual information within a large receptive field. (2) Instead of using a fixed kernel for all samples (e.g. convolution and deconvolution), CARAFE++ generates adaptive kernels on-the-fly to enable instance-specific content-aware handling. (3) CARAFE++ introduces little computational overhead and can be readily integrated into modern network architectures. We conduct comprehensive evaluations on standard benchmarks in object detection, instance/semantic segmentation and image inpainting. CARAFE++ shows consistent and substantial gains across all the tasks (2.5% APbox, 2.1% APmask, 1.94% mIoU, 1.35 dB respectively) with negligible computational overhead.

Introduction

In modern convolutional neural networks (CNNs), "feature reassembly"—the process of upsampling or downsampling feature maps—is a fundamental operation. Whether enlarging a feature map for pixel-level prediction or shrinking it to increase the receptive field, the choice of reassembly method significantly impacts performance.

Traditional methods generally fall into two categories, both with inherent limitations:

  1. Rule-based (Pooling/Interpolation): Methods like Max Pooling or Bilinear Interpolation follow fixed mathematical rules regardless of the image content. They fail to capture rich semantic information.
  2. Learning-based (Conv/Deconv): Standard convolutions use learned weights that are fixed across the entire image. This "one-size-fits-all" approach cannot adapt to local variations, and increasing kernel sizes to capture more context becomes computationally prohibitive.

CARAFE++ bridges this gap by providing a "smart" operator that is both content-aware and highly efficient.

The CARAFE++ Architecture

CARAFE++ consists of two primary modules that work in tandem to resize feature maps dynamically based on local context.

The CARAFE++ framework consists of a Kernel Prediction Module and a Content-aware Reassembly Module.

1. Kernel Prediction Module

This module acts as the "brain" of the operator. For every target location in the output map, it predicts a unique reassembly kernel through three steps:

  • Channel Compressor: It reduces input feature channels (e.g., from 256 to 64) to minimize computational load.
  • Content Encoder: A small convolution analyzes the local neighborhood to generate specific kernel weights.
  • Kernel Normalizer: A softmax function ensures kernel weights sum to one, maintaining stable feature values.

2. Content-aware Reassembly Module

This module applies the custom-predicted kernels to the input features, performing a weighted sum of pixels in a local neighborhood. Because kernels are content-dependent, the network can selectively "attend" to relevant pixels (such as object boundaries) while ignoring background noise. CARAFE++ is "unified" because the same architecture handles both upsampling and downsampling by adjusting stride and kernel mapping.

Experimental Results

CARAFE++ was evaluated across several standard computer vision benchmarks, including MS COCO 2017 (detection/segmentation), ADE20k (semantic segmentation), and Places (inpainting). The performance gains were consistent and substantial:

  • Object Detection: Improved Faster R-CNN baseline by 2.5% AP on COCO.
  • Instance Segmentation: Boosted Mask R-CNN by 2.1% AP.
  • Semantic Segmentation: Increased UperNet performance by 1.94% mIoU on ADE20k.
  • Image Inpainting: Improved results by 1.35 dB PSNR on the Places dataset.

Despite these gains, the computational cost remains minimal. For Faster R-CNN, inference speed only slightly decreased from 10.3 fps to 9.9 fps, demonstrating its practical utility.

Content-Aware Visualization and Insights

A key strength of CARAFE++ is its ability to perform instance-specific handling. Unlike standard convolutions that learn one filter for all images, CARAFE++ creates a new filter for every patch of every image, adapting to the specific geometry of observed objects.

Visualization of reassembled units. Red dots indicate the pixels that contribute most to the reassembly of the green center point.

As shown in the visualization above, when the operator reassembles a feature at a specific point (green dot), the predicted kernels automatically focus on related pixels (red dots) belonging to the same object rather than the background. This effectively allows the network to "group" related features during the resizing process. Furthermore, by using a channel compressor, CARAFE++ can afford a large receptive field (e.g., 5x5 or 7x7) without the parameter explosion typical of large-kernel convolutions.

Conclusion and Future Work

CARAFE++ serves as a powerful "plug-and-play" module that can replace standard Max Pool or Bilinear Upsampling in architectures like ResNet or Feature Pyramid Networks (FPN) for immediate performance boosts.

While highly effective, current limitations include a focus on integer scaling factors (e.g., 2x, 4x). Future research directions include supporting arbitrary non-integer scale factors, exploring integration with Neural Architecture Search (NAS), and investigating benefits for low-level vision tasks such as image denoising or deblurring.

Summary updated

Abstract

Feature reassembly, i.e. feature downsampling and upsampling, is a key operation in a number of modern convolutional network architectures, e.g., residual networks and feature pyramids. Its design is critical for dense prediction tasks such as object detection and semantic/instance segmentation. In this work, we propose unified Content-Aware ReAssembly of FEatures (CARAFE++), a universal, lightweight and highly effective operator to fulfill this goal. CARAFE++ has several appealing properties: (1) Unlike conventional methods such as pooling and interpolation that only exploit sub-pixel neighborhood, CARAFE++ aggregates contextual information within a large receptive field. (2) Instead of using a fixed kernel for all samples (e.g. convolution and deconvolution), CARAFE++ generates adaptive kernels on-the-fly to enable instance-specific content-aware handling. (3) CARAFE++ introduces little computational overhead and can be readily integrated into modern network architectures. We conduct comprehensive evaluations on standard benchmarks in object detection, instance/semantic segmentation and image inpainting. CARAFE++ shows consistent and substantial gains across all the tasks (2.5% APbox, 2.1% APmask, 1.94% mIoU, 1.35 dB respectively) with negligible computational overhead. It shows great potential to serve as a strong building block for modern deep networks.
BibTeX
@article{wang2022carafe,
  author = {Jiaqi Wang and Kai Chen and Rui Xu and Ziwei Liu and Chen Change Loy and Dahua Lin},
  title = {{CARAFE++: Unified Content-Aware ReAssembly of FEatures}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2022},
  url = {https://openreview.net/forum?id=TvsCBpbG9Vn},
}
Figure