Efficient and Invariant Convolutional Neural Networks for Dense Prediction

Hongyang Gao, Shuiwang Ji

Introduction

Standard Convolutional Neural Networks (CNNs) are inherently translation-invariant, meaning they can recognize patterns regardless of their position in an image. However, they struggle with rotations and flips. In many "dense prediction" tasks, such as medical image segmentation, the orientation of features like cell boundaries is arbitrary.

The traditional solution is data augmentation, where models are trained on rotated and flipped versions of images. While effective, this often requires a test-time "ensemble" approach—running the model eight times on different orientations of the same image—which is computationally expensive and memory-intensive for high-resolution tasks.

Proposed Method: Invariant Convolutional Layers

The authors propose a novel architecture that achieves rotation and flip invariance by transforming the convolutional layers themselves rather than the input data.

Kernel Transformation

Instead of transforming massive internal feature maps, the method transforms the small $3 \times 3$ convolutional kernels. A single learned kernel can be rotated into eight discrete orientations (0°, 45°, 90°, etc.) and flipped horizontally or vertically. This allows the network to learn a single set of weights that applies across all possible orientations of a feature.

Illustration of 8 discrete rotations for a 3x3 convolutional kernel, enabling the network to detect features at various angles.

Efficient Invariance via Maxout

Simply rotating kernels would normally octuple (8x) the number of feature maps, leading to a "memory explosion." To solve this, the authors apply a Maxout operation. This operation takes the maximum value across the transformed feature maps at each pixel location, collapsing them back into a single map. This retains only the strongest "invariant" signal, keeping memory usage manageable while baking invariance directly into the architecture.

Architecture of the proposed rotation and flip convolution layers. Kernels are transformed, applied to feature maps, and then compressed using a Maxout operation.

Evaluation and Results

The model was evaluated on the CREMI (Circuit Reconstruction from Electron Microscopy Images) dataset, which involves segmenting neuronal boundaries in brain images—a task where rotation invariance is critical.

Key Performance Metrics

The proposed model was compared against a baseline CNN with and without traditional data augmentation:

  • Accuracy (AUC): The proposed model achieved an AUC of 0.941, outperforming the baseline with heavy data augmentation (0.939) and the baseline without augmentation (0.931).
  • Inference Speed:
    • Baseline (no augmentation): 28 minutes.
  • Proposed Model: 45 minutes.
    • Baseline (augmentation ensemble): 226 minutes.

By integrating invariance into the kernels, the model only needs to process the image once to achieve results superior to an 8-pass ensemble, representing a 5x speedup at test time.

Conclusion

This research demonstrates that kernel-level transformations combined with Maxout compression can provide robust rotation and flip invariance without the massive computational overhead of traditional methods. This approach is particularly valuable for dense prediction tasks in medical imaging, where both accuracy and processing efficiency are paramount.

Summary updated

BibTeX
@inproceedings{gao2017efficient,
  author = {Hongyang Gao and Shuiwang Ji},
  title = {{Efficient and Invariant Convolutional Neural Networks for Dense Prediction}},
  booktitle = {ICDM},
  year = {2017},
  url = {https://openreview.net/forum?id=XWiUiBvL4s5},
}
Figure