Receptive Fields As Experts in Convolutional Neural Architectures

Dongze Lian, Weihao Yu, Xinchao Wang

TL;DR

The paper introduces Mixture of Receptive Fields (MoRF), a novel architectural concept that treats different convolutional kernel sizes as "experts" dynamically selected based on the input image. By leveraging a re-parameterization trick, MoRF allows models to benefit from a diverse ensemble of receptive fields during training while merging them into a single, fast convolutional kernel for inference. This boosts accuracy across vision tasks with negligible speed penalties.

The Problem: Fixed Receptive Fields

In Convolutional Neural Networks (CNNs), the "receptive field"—the specific area of an input image that a single neuron can "see"—is a fundamental design choice. Historically, this has been a manual tug-of-war: VGGNet popularized small 3x3 kernels for efficiency, while recent architectures like ConvNeXt use larger 7x7 kernels to capture more global context.

However, a fixed kernel size is a compromise. Different images (and different parts of the same image) contain information at different scales. A close-up of a texture might benefit from a small, local receptive field, while a wide landscape requires a larger one to understand spatial relationships. Manually tuning these sizes for every layer is nearly impossible. The authors aim to make the receptive field dynamic and input-dependent without the massive computational overhead usually associated with complex ensembles.

Methods: The MoRF Layer

The authors propose the MoRF layer, which replaces a standard convolutional layer with a "mixture" of experts.

MoRF Layer Architecture
  1. Receptive Field Experts: Instead of one convolution, the layer contains multiple parallel branches with different kernel sizes (e.g., 3x3, 5x5, 7x7, up to 15x15).
  2. Routing Mechanisms: To decide which expert to use for a given input, they introduce an Attention Module. This module looks at the input features and generates "routing weights."
    • Hard Routing: Selects only the top-$k$ most relevant experts (similar to traditional Mixture of Experts).
  3. Soft Routing: Computes a weighted sum of all experts, providing a more nuanced feature map.
  4. Kernel Fusion (The Inference Trick): The primary innovation for efficiency is the use of re-parameterization. Since convolutions are linear operations, multiple parallel kernels of different sizes can be mathematically merged into a single, larger kernel after training.
ConvNeXt Integration and Inference

As illustrated above, this re-parameterization allows the model to act like a complex mixture during training (b and c) but run as a single, standard convolution during deployment (d).

Datasets

The researchers evaluated MoRF across three standard computer vision benchmarks:

  • ImageNet-1K: For image classification.
  • MS COCO: For object detection and instance segmentation (using Mask R-CNN and Cascade Mask R-CNN frameworks).
  • ADE20K: For semantic segmentation (using the UperNet framework).

Key Insights and Ideas

  • Dynamic Context Selection: MoRF allows the network to "choose its own adventure." If an image requires global context, the routing mechanism weights the larger kernels more heavily; if it needs fine detail, it leans on the smaller ones.
  • The "Randomness" of Selection: Interestingly, the authors found that expert selection across layers doesn't follow a strict "small-to-large" hierarchy. Visualizations showed that selection is relatively varied across different blocks, suggesting that the optimal receptive field is highly specific to the individual input and layer depth.
  • Breaking the Efficiency Trade-off: Usually, adding "experts" or branches increases the parameter count and slows down the model. MoRF bypasses this during inference by "collapsing" the branches back into one, proving that we can have architectural complexity during learning without paying for it during execution.
  • Soft vs. Hard Routing: While hard routing is more common in large-scale language models to save compute, the authors found that soft routing generally performed better for vision tasks because it preserves a richer set of feature representations.

Key Results

MoRF consistently outperformed baseline models (ResNet and ConvNeXt) across all tasks:

  • Image Classification: Integrating MoRF into ResNet-18 boosted Top-1 accuracy on ImageNet from 70.4% to 74.6% (+4.2%) with a minimal drop in throughput. For ConvNeXt-T, accuracy rose from 82.1% to 82.6%.
  • Object Detection: On the COCO dataset, MoRF improved the Box AP of ConvNeXt-T from 46.2 to 46.8.
  • Semantic Segmentation: On ADE20K, MoRF improved the mIoU of a ConvNeXt-T based UperNet from 46.7 to 47.7.
  • Inference Speed: Thanks to re-parameterization, the inference speed of MoRF-enhanced models remained comparable to their single-kernel counterparts (e.g., ResNet-18 baseline at 1643 imgs/s vs. MoRF-soft at 1402 imgs/s).

Limitations and Future Work

  • Training Resources: While inference is fast, the training phase remains resource-intensive. Because the model must compute all expert branches before they are merged, it requires more memory and time during the training cycle.
  • Expert Scaling: Due to hardware constraints, the authors limited their experiments to a maximum of seven experts. They note that scaling to a much larger number of experts (as seen in NLP models with hundreds of experts) remains a challenge for future work.
  • Application to Transformers: The current work focuses on CNNs. The authors suggest that the MoRF concept—specifically the idea of dynamic, input-dependent "windows" of attention—could be adapted to Vision Transformers (ViTs) to further optimize their spatial processing.

Summary updated

Abstract

The size of spatial receptive fields, from the early 3$\times$3 convolutions in VGGNet to the recent 7$\times$7 convolutions in ConvNeXt, has always played a critical role in architecture design. In this paper, we propose a Mixture of Receptive Fields (MoRF) instead of using a single receptive field. MoRF contains the combinations of multiple receptive fields with different sizes, e.g., convolutions with different kernel sizes, which can be regarded as experts. Such an approach serves two functions: one is to select the appropriate receptive field according to the input, and the other is to expand the network capacity. Furthermore, we also introduce two types of routing mechanisms, hard routing and soft routing to automatically select the appropriate receptive field experts. In the inference stage, the selected receptive field experts are merged via re-parameterization to maintain a similar inference speed compared to the single receptive field. To demonstrate the effectiveness of MoRF, we integrate the MoRF concept into multiple architectures, e.g., ResNet and ConvNeXt. Extensive experiments show that our approach outperforms the baselines in image classification, object detection, and segmentation tasks without significantly increasing the inference time.
BibTeX
@inproceedings{lian2024receptive,
  author = {Dongze Lian and Weihao Yu and Xinchao Wang},
  title = {{Receptive Fields As Experts in Convolutional Neural Architectures}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=HGSIpeNNfM},
}
Figure