Accurate Shift Invariant Convolutional Neural Networks Using Gaussian-Hermite Moments

Jaspreet Singh, Petra Bosilj, Grzegorz Cielniak

Abstract

The convolutional neural networks (CNNs) are not inherently shift invariant or equivariant. The downsampling operation, used in CNNs, is one of the key reasons which breaks the shift invariant property of a CNN. Conversely, downsampling operation is important to improve computational efficiency and increase the area of the receptive field for more contextual information. In this work, we propose Gaussian-Hermite Sampling (GHS), a novel downsampling strategy designed to achieve accurate shift invariance. GHS leverages Gaussian-Hermite polynomials to perform shift-consistent sampling, enabling CNN layers to maintain invariance to arbitrary spatial shifts prior to training. When integrated into standard CNN architectures, the proposed method embeds shift invariance directly at the layer level without requiring architectural modifications or additional training procedures. We evaluate the proposed approach on CIFAR-10, CIFAR-100, and MNIST-rot datasets. Experimental results demonstrate that GHS significantly improves shift consistency, achieving 100% classification consistency under spatial shifts, while also improving classification accuracy compared to baseline CNN models.

Introduction

A fundamental expectation in computer vision is shift invariance: the ability of a model to recognize an object regardless of its spatial position. While convolution operations are naturally equivariant—meaning feature maps shift in tandem with the input—standard downsampling operations such as max-pooling or strided convolutions frequently break this property. This paper introduces Gaussian-Hermite Sampling (GHS), a parameter-free downsampling method that achieves 100% classification consistency under spatial shifts without requiring extra training or data augmentation.

The Downsampling Dilemma

Downsampling is essential for computational efficiency and expanding receptive fields, yet it introduces aliasing. When an input shifts by a single pixel, the downsampling grid may align differently, leading to divergent feature representations. This sensitivity means that minor translations can cause a CNN to flip its prediction entirely. While existing solutions like anti-aliasing (BlurPool) or Adaptive Polyphase Sampling (APS) mitigate these effects, they often fail to provide mathematical guarantees of perfect invariance or sacrifice discriminative detail.

Gaussian-Hermite Sampling (GHS)

The proposed GHS method treats local image patches as signals to be decomposed into orthogonal basis functions. The process involves four key components:

  1. Gaussian-Hermite Polynomials (GHPs): These orthogonal functions are windowed by a Gaussian curve, providing a built-in low-pass filter that naturally prevents aliasing.
  2. Discrete Moments: The image is projected onto GHPs to calculate moments—mathematical descriptors that summarize local content.
  3. The Max-Centered Pivot: To ensure true invariance, GHS defines a "pivot position" based on the maximum activation in the feature map. By anchoring the coordinate system of the moments to this pivot, the representation becomes invariant to circular translations (wrap-shifts).
  4. Reconstruction: The image is reconstructed at a lower resolution from these moments. This parameter-free process allows GHS to be integrated into existing architectures like ResNet without additional training.

Key Insights: Invariance vs. Equivariance

Understanding the distinction between invariance and equivariance is critical for robust CNN design. Invariance implies that the output remains unchanged despite input transformations, whereas equivariance implies that the output transforms in a predictable way relative to the input.

Comparison of Invariance and Equivariance in Image Processing

GHS provides exact layer-wise shift invariance by anchoring the coordinate system of the moments to the strongest signal in a patch. This ensures that even if the input moves, the "viewpoint" of the moments moves with it, maintaining a consistent output representation.

Experimental Results

The researchers evaluated GHS on CIFAR-10, CIFAR-100, and MNIST-rot benchmarks, as well as the classic "Cameraman" image for numerical analysis.

  • 100% Consistency: GHS achieved perfect classification consistency across all tested datasets, ensuring identical predictions regardless of input shifts.
  • Accuracy Improvements: Beyond robustness, GHS improved raw performance. On CIFAR-10 with a ResNet-18 backbone, GHS reached 95.54% accuracy, significantly outperforming the standard baseline (91.96%) and APS (94.48%).
  • Out-of-Distribution Robustness: GHS maintained 100% consistency under random patch erasures and vertical flips, demonstrating superior generalization compared to standard data augmentation.

Conclusion and Future Work

Gaussian-Hermite Sampling offers a mathematically rigorous solution to the shift-invariance problem in CNNs. While GHS is currently more computationally intensive than standard max-pooling (approximately 10x slower), its performance is comparable to other advanced sampling methods like APS. Future research will focus on optimizing these calculations through recurrence relations to avoid expensive factorials and extending the framework to tasks requiring precise localization, such as object detection and semantic segmentation.

Summary updated

Abstract

The convolutional neural networks (CNNs) are not inherently shift invariant or equivariant. The downsampling operation, used in CNNs, is one of the key reasons which breaks the shift invariant property of a CNN. Conversely, downsampling operation is important to improve computational efficiency and increase the area of the receptive field for more contextual information. In this work, we propose Gaussian-Hermite Sampling (GHS), a novel downsampling strategy designed to achieve accurate shift invariance. GHS leverages Gaussian-Hermite polynomials to perform shift-consistent sampling, enabling CNN layers to maintain invariance to arbitrary spatial shifts prior to training. When integrated into standard CNN architectures, the proposed method embeds shift invariance directly at the layer level without requiring architectural modifications or additional training procedures. We evaluate the proposed approach on CIFAR-10, CIFAR-100, and MNIST-rot datasets. Experimental results demonstrate that GHS significantly improves shift consistency, achieving 100% classification consistency under spatial shifts, while also improving classification accuracy compared to baseline CNN models.
BibTeX
@misc{singh2026accurate,
  author = {Jaspreet Singh and Petra Bosilj and Grzegorz Cielniak},
  title = {{Accurate Shift Invariant Convolutional Neural Networks Using Gaussian-Hermite Moments}},
  year = {2026},
  eprint = {2603.17098},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.17098},
}
Figure