CARAFE: Content-Aware ReAssembly of FEatures

Jiaqi Wang, Kai Chen, Rui Xu, Ziwei Liu, Chen Change Loy, Dahua Lin

Abstract

Feature upsampling is a fundamental operation in modern deep learning architectures, particularly for dense prediction tasks such as object detection and semantic segmentation. Traditional methods like bilinear interpolation and deconvolution often fall short due to their content-agnostic nature or high computational costs. This paper introduces CARAFE (Content-Aware ReAssembly of FEatures), a lightweight and universal upsampling operator. CARAFE adaptively generates kernels based on local image content, enabling a large receptive field with negligible overhead. Experimental results across multiple benchmarks demonstrate that CARAFE consistently outperforms standard upsampling methods, providing significant gains in accuracy for a variety of vision tasks.

Introduction

In dense prediction tasks, models must output high-resolution maps to identify pixels or fill missing information. Feature upsampling is the core mechanism for restoring spatial resolution. Currently, two main approaches dominate:

  1. Interpolation (Nearest Neighbor/Bilinear): These methods are computationally efficient but rely on fixed geometric formulas. They fail to account for the semantic content of the image, treating smooth regions and complex textures identically.
  2. Deconvolution: While learnable, deconvolution applies the same kernel across the entire image, ignoring local variations. It is also computationally intensive and often restricted by a limited receptive field.

CARAFE addresses these limitations by providing a "content-aware" solution that adapts to the image, maintains a large receptive field, and remains lightweight enough for real-time applications.

Methodology

The CARAFE operator decomposes the upsampling process into two primary modules: the Kernel Prediction Module and the Content-Aware Reassembly Module.

The CARAFE architecture: The Kernel Prediction Module generates instance-specific kernels, which the Reassembly Module then applies to the input features.
  1. Kernel Prediction Module: This module predicts a unique upsampling kernel for every location in the input feature map. It consists of a Channel Compressor to reduce input dimensions, a Content Encoder to analyze local context, and a Kernel Normalizer to ensure stable feature values.
  2. Content-Aware Reassembly Module: This module applies the predicted kernels to the original features. Unlike standard convolutions that transform features across channels, CARAFE "reassembles" spatial pixels within a local neighborhood, making the process highly efficient.

Key Insights

A core innovation of CARAFE is the shift from feature transformation to feature reassembly. By focusing purely on spatial reorganization, CARAFE avoids the heavy computational burden of traditional learnable upsamplers.

Comparison of reassembly regions: CARAFE (right) adapts its sampling locations to the object's shape, whereas standard interpolation (left) uses a fixed grid.

Furthermore, CARAFE leverages a larger context (e.g., 5x5 or 7x7) compared to the 2x2 neighborhood used in bilinear interpolation. This allows the model to make better-informed decisions about how to reconstruct high-resolution details. The authors also found that predicting kernels from compressed feature channels does not significantly impact performance, allowing for a very lightweight implementation.

Experimental Results

CARAFE was evaluated on several standard benchmarks, including MS COCO 2017, ADE20k, and Places. It consistently provided "plug-and-play" improvements:

  • Object Detection: Improved Faster R-CNN by 1.2% AP on COCO.
  • Instance Segmentation: Boosted Mask R-CNN by 1.3% AP.
  • Semantic Segmentation: Enhanced UperNet by 1.8% mIoU on ADE20k.
  • Image Inpainting: Increased PSNR by 1.1 dB.

In terms of efficiency, CARAFE is significantly lighter than deconvolution. For a 2x upsampling of a 256-channel map, CARAFE requires approximately 199k FLOPs, compared to 1180k FLOPs for deconvolution. Qualitatively, CARAFE produces sharper boundaries by recognizing object shapes and aligning kernels accordingly.

Conclusion

CARAFE represents a significant step forward in feature upsampling, offering a universal, content-aware, and efficient alternative to traditional methods. While its success in high-level vision tasks is well-documented, future research may explore its application in low-level vision tasks like extreme super-resolution. Additionally, optimizing custom CUDA kernels could further enhance its real-world speed on specialized hardware.

Summary updated

Abstract

Feature upsampling is a key operation in a number of modern convolutional network architectures, e.g. feature pyramids. Its design is critical for dense prediction tasks such as object detection and semantic/instance segmentation. In this work, we propose Content-Aware ReAssembly of FEatures (CARAFE), a universal, lightweight and highly effective operator to fulfill this goal. CARAFE has several appealing properties: (1) Large field of view. Unlike previous works (e.g. bilinear interpolation) that only exploit sub-pixel neighborhood, CARAFE can aggregate contextual information within a large receptive field. (2) Content-aware handling. Instead of using a fixed kernel for all samples (e.g. deconvolution), CARAFE enables instance-specific content-aware handling, which generates adaptive kernels on-the-fly. (3) Lightweight and fast to compute. CARAFE introduces little computational overhead and can be readily integrated into modern network architectures. We conduct comprehensive evaluations on standard benchmarks in object detection, instance/semantic segmentation and inpainting. CARAFE shows consistent and substantial gains across all the tasks (1.2%, 1.3%, 1.8%, 1.1db respectively) with negligible computational overhead. It has great potential to serve as a strong building block for future research. It has great potential to serve as a strong building block for future research. Code and models are available at https://github.com/open-mmlab/mmdetection.
BibTeX
@inproceedings{wang2019carafe,
  author = {Jiaqi Wang and Kai Chen and Rui Xu and Ziwei Liu and Chen Change Loy and Dahua Lin},
  title = {{CARAFE: Content-Aware ReAssembly of FEatures}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00310},
}
Figure