Receptive Field Size Optimization With Continuous Time Pooling

Dora Babicz, Soma Kontar, Mark Peto, Andras Fulop, Gergely Szabo, Andras HorvathWACV 2021

TL;DR

This paper introduces "continuous time pooling," a novel pooling operation for Convolutional Neural Networks (CNNs) that mimics biological receptive fields by using a diffusion process. This approach allows for learnable, continuous receptive field sizes, leading to improved accuracy and robustness compared to traditional discrete pooling methods like max pooling.

What Problem Does This Paper Address?

Convolutional Neural Networks (CNNs) rely heavily on pooling operations to reduce dimensionality and introduce invariance to small perturbations. However, standard pooling methods like max pooling have limitations: they are discrete, making their kernel sizes difficult to optimize as hyperparameters, and they can introduce quantization errors and a sparse update mechanism that neglects positional information within the receptive field. This can lead to issues like aliasing effects and a lack of shift equivariance. Biological receptive fields, on the other hand, are often continuous and sensitive to the precise location of stimuli, suggesting that a more nuanced pooling approach could benefit CNNs.

Methods

The core contribution is "continuous time pooling," which replaces discrete pooling with a continuous-time differential equation that models a non-linear diffusion process. This process effectively allows activations to diffuse and merge, with the "pooling strength" parameter (Ps) controlling the extent of this diffusion, analogous to the size and shape of a receptive field. This continuous process is then approximated numerically using the Euler method, making it implementable on standard hardware. Crucially, the pooling strength parameter (Ps) can be optimized using gradient descent, allowing the network to learn optimal receptive field sizes for specific tasks. The authors also propose a method to "quantize" these learned continuous receptive fields back into discrete maximum pooling operations for inference, balancing accuracy gains with computational efficiency.

Datasets

  • Simulated Dataset: Used to demonstrate weaknesses of maximum pooling and the benefits of the proposed method.
  • MNIST: Used for digit classification with LeNet-5 architecture.
  • CIFAR-10: Used for image classification with AlexNet architecture.
  • ImageNet 2012: Used for image classification with VGG-16, DenseNets, and ResNet architectures.
  • MS-COCO: Used for instance segmentation and object detection with Mask R-CNN and ResNeXt-101 backbone.

Key Insights and Ideas

  • Biological Inspiration: The continuous nature of biological receptive fields provides a strong motivation for developing more flexible pooling mechanisms.
  • Learnable Receptive Fields: Allowing receptive field size to be a continuous, learnable parameter (via Ps) offers significant advantages over fixed, discrete kernel sizes.
  • Positional Sensitivity: Unlike max pooling, continuous pooling inherently considers the location of activations within the receptive field, leading to more nuanced feature extraction.
  • Quantization for Efficiency: The ability to quantize learned continuous receptive fields back to discrete operations is a clever way to retain accuracy gains while mitigating computational overhead during inference.
  • Addressing Shift Equivariance: The authors hypothesize that continuous pooling can help address issues of shift equivariance in CNNs.

Key Results

  • Pixel Distance Task: Continuous pooling achieved a mean squared error of 10, significantly outperforming max pooling's error of 70 on a simulated pixel distance estimation task.
  • MNIST: Achieved 98.87% accuracy with LeNet-5, compared to 95.30% for max pooling.
  • CIFAR-10: Showed a 3.56% increase in classification accuracy over other pooling variants with AlexNet.
  • ImageNet: Continuous pooling resulted in the highest top-1 accuracy for VGG-16, DenseNet-121, and ResNet-50 architectures.
  • MS-COCO: Continuous pooling showed improved performance in instance segmentation and object detection tasks.
  • Computational Cost: While continuous pooling increases computational complexity during training (up to 87% increase in wall time for VGG-16), the quantized version offers a balance, with only a minor drop in accuracy.

Limitations and Future Work

  • Computational Overhead: The iterative nature of the continuous pooling approximation increases computational cost during training. The paper addresses this by proposing a quantization method for inference.
  • Hyperparameter Tuning: While Ps is learnable, the number of iterations (N) for the Euler approximation is still a hyperparameter.
  • Future Work: The authors plan to investigate layer- and neuron-dependent Ps parameters, explore combinations with other pooling methods (like LIP), and further analyze the impact of upscaling on image features. They also suggest that the learned receptive field sizes could be used to inform the design of network architectures.

Summary updated

BibTeX
@inproceedings{babicz2021receptive,
  author = {Dora Babicz and Soma Kontar and Mark Peto and Andras Fulop and Gergely Szabo and Andras Horvath},
  title = {{Receptive Field Size Optimization With Continuous Time Pooling}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}

Loading related content...

Figure