Contextual Convolutional Networks

Shuxian Liang, Xu Shen, Tongliang Liu, Xian-Sheng HuaICLR 2023 · OpenReview

Contextual Convolutional Networks (Contextual CNN) break the traditional "representation-then-classification" paradigm by using potential category memberships as priors to guide feature extraction. By predicting likely classes early and using that "context" to modulate convolution weights and sampling locations, this architecture achieves a significant +1.8% top-1 accuracy boost on ImageNet over the state-of-the-art ConvNeXt-Tiny backbone.

What Problem Does This Paper Address?

Most modern Convolutional Neural Networks (CNNs) follow a strictly bottom-up, category-agnostic approach. They pass an image through dozens of layers of static filters to generate a universal representation. Only at the very last layer does a classifier decide if the image contains a "husky" or a "wolf."

The authors argue this is inefficient and unlike the human visual system. In neuroscience, "top-down modulation" allows our brains to use high-level context to influence how we process low-level sensory input. If you think you see a dog, your brain proactively looks for specific features like ears or fur patterns. Standard CNNs lack this feedback loop, often struggling to distinguish between semantically similar classes that share many low-level features.

Methods: The Contextual Convolutional Block

The authors introduce a novel "Contextual Convolutional" block that can be dropped into existing architectures like ResNet or ConvNeXt. The system works through three main components:

  1. Contextualizing Layers: At various stages of the network, the model performs an intermediate "classification" step. It projects current features into an embedding space and identifies the top-k most likely classes.
  2. Merging: These top-k class embeddings are merged into a single contextual prior vector. This vector represents the "best guess" of what the object is at that stage.
  3. Modulation (The Core Innovation): This prior vector is used to dynamically adjust the convolution operation in the next stage in two ways:
  4. Sampling Offsets ($\Delta p$): It tells the network where to look by deforming the convolution's sampling grid (similar to Deformable Convolutions, but guided by class context).
  5. Weight Offsets ($\Delta w$): It tells the network how to look by adjusting the actual weights of the convolutional filters to focus on class-specific discriminative features.
Overall architecture of the Contextual Convolutional Network.

The network is trained end-to-end with a joint loss function that supervises both the final output and the intermediate "context-generating" classifications.

Key Insights and Ideas

  • Progressive Refinement: The network reduces the number of candidate classes stage-by-stage (e.g., 1000 $\to$ 500 $\to$ 200 $\to$ 50 $\to$ 1). This mimics a "coarse-to-fine" recognition process.
  • Context as a Filter: The most striking finding is that "knowing" the potential class helps the model ignore confusing shared features. For example, while a standard model might get confused between a "pier" and a "boathouse" because both involve water and wood, the Contextual CNN uses the "pier" prior to specifically look for the long walkway stretching into the water.
Grad-CAM comparison showing how contextual priors help the model focus on discriminative features (e.g., the walkway of a pier).
  • Efficiency: Despite the extra classification and modulation steps, the authors managed to keep the computational overhead low. The Contextual ConvNeXt-T adds only about 3% to the FLOPs (floating-point operations) while delivering a nearly 2% accuracy gain.

Key Results

The model was evaluated on several industry-standard benchmarks, consistently outperforming its "plain" counterparts across all tasks:

  • ImageNet-1K (Classification): Contextual ConvNeXt-Tiny reached 83.9% top-1 accuracy, a +1.8% improvement over the base ConvNeXt-T.
  • Kinetics-400 (Video Action Recognition): Achieved 77.0% top-1 accuracy, outperforming ResNet-101 despite having significantly fewer parameters and FLOPs.
  • COCO (Object Detection & Instance Segmentation): Improved Box AP by +1.1 and Mask AP by +1.0 over the ConvNeXt-T baseline, proving that the contextual priors learned for classification also translate well to localization and segmentation.

Limitations and Future Work

While highly effective, the architecture is more complex to implement than standard backbones due to the intermediate embedding layers and modulation blocks. Furthermore, its performance relies heavily on high-quality class embeddings. The authors used a fixed embedding layer; exploring dynamic or learned embeddings from multi-modal models (like CLIP) could be a fruitful next step. Finally, while the FLOP count is low, dynamic operations like sampling offsets often see a larger "real-world" latency hit on GPUs than standard convolutions, though the authors note their throughput remains highly competitive.

Summary updated

Abstract

This paper presents a new Convolutional Neural Network, named Contextual Convolutional Network, that capably serves as a general-purpose backbone for visual recognition. Most existing convolutional backbones follow the representation-to-classification paradigm, where representations of the input are firstly generated by category-agnostic convolutional operations, and then fed into classifiers for specific perceptual tasks (e.g., classification and segmentation). In this paper, we deviate from this classic paradigm and propose to augment potential category memberships as contextual priors in the convolution for contextualized representation learning. Specifically, top-k likely classes from the preceding stage are encoded as a contextual prior vector. Based on this vector and the preceding features, offsets for spatial sampling locations and kernel weights are generated to modulate the convolution operations. The new convolutions can readily replace their plain counterparts in existing CNNs and can be easily trained end-to-end by standard back-propagation without additional supervision. The qualities of Contextual Convolutional Networks make it compatible with a broad range of vision tasks and boost the state-of-the-art architecture ConvNeXt-Tiny by 1.8% on top-1 accuracy of ImageNet classification. The superiority of the proposed model reveals the potential of contextualized representation learning for vision tasks. Code is available at: \url{https://github.com/liang4sx/contextual_cnn}.
BibTeX
@inproceedings{liang2023contextual,
  author = {Shuxian Liang and Xu Shen and Tongliang Liu and Xian-Sheng Hua},
  title = {{Contextual Convolutional Networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=PldynS56bN},
}
Figure