Substituting Convolutions for Neural Network Compression

Elliot J. Crowley, Gavin Gray, Jack Turner, Amos Storkey

Introduction

Deep Neural Networks (DNNs) have achieved remarkable success in computer vision but remain computationally expensive, often requiring millions or even billions of parameters. This resource intensity poses a significant challenge for deployment on "edge" devices such as smartphones, wearables, and embedded systems. While pruning and quantization are common compression techniques, they often require extensive hyperparameter tuning and are architecture-specific.

This research addresses the gap by focusing on the most parameter-heavy components of modern networks: 3×3 convolutions and 1×1 (pointwise) convolutions. The authors propose a general, "minimal tuning" strategy to replace these expensive operations with efficient mathematical substitutes without redesigning the entire network architecture from scratch.

Methodology: Substitute and Distill

The proposed workflow, termed "Substitute and Distill," involves a two-step process to reduce model size while maintaining performance.

1. Substitution

A pre-trained "teacher" network is transformed into a "student" network by replacing its standard convolutional layers with cheaper alternatives:

  • 3×3 Convolutions: These are replaced with grouped convolutions, bottleneck blocks, or a hybrid "Bottleneck Grouped" (BG) approach. Grouped convolutions restrict channel mixing, significantly reducing the number of parameters.
  • 1×1 Convolutions: Since pointwise convolutions are equivalent to fully-connected layers applied per pixel, they are replaced with efficient linear algebra transforms such as Tensor-Train (TT), Tucker Decomposition, HashedNet, and ACDC (a structured transform based on Discrete Cosine Transforms).
Visualization of grouped convolutions as a substitute for standard 3x3 convolutions.

2. Distillation and Regularization

To recover accuracy lost during substitution, the student network is trained using Knowledge Distillation (KD) or Attention Transfer (AT) from the original teacher. This allows the smaller model to learn the complex feature representations of the larger one.

A critical technical contribution is Compression Ratio Scaled (CRS) weight decay. Standard regularization often fails for compressed layers. The authors derived a scaling factor based on the compression ratio to stabilize training and maintain the total variance of the weights.

Key Insights

  • Architecture Preservation: Maintaining the original network's depth and width while swapping internal operations is often more effective than simply making the network shallower or thinner.
  • The 1×1 Opportunity: Pointwise convolutions dominate the parameter count in modern architectures like MobileNet. Treating these as linear algebra compression problems opens up a vast library of efficient transforms that the computer vision community has largely overlooked.
  • Distillation as Glue: Substitute networks perform poorly when trained from scratch but reach near-original accuracy when "taught" by a full-sized network.

Experimental Results

The approach was evaluated on CIFAR-10, CIFAR-100, and ImageNet benchmarks:

  • CIFAR-10: A WideResNet-40-2 was compressed by 5× with only a 0.27% drop in accuracy using grouped convolutions.
  • ImageNet: Compressing a ResNet-34 by 2.65× actually resulted in a slight accuracy increase, suggesting that the distillation process serves as a powerful regularizer.
  • Pareto Optimality: The proposed substitutions pushed the Pareto frontier for architectures like DARTS and MobileNetV2, providing superior accuracy-per-parameter compared to many manually designed efficient models.

Conclusion and Future Work

The "Substitute and Distill" strategy provides a robust framework for model compression with minimal manual tuning. While the reduction in parameters and operations is clear, actual runtime performance varies by hardware. Future research may explore automated search techniques to "mix and match" different substitutions within a single network for optimal efficiency.

Summary updated

Abstract

Many practitioners would like to deploy deep, convolutional neural networks in memory-limited scenarios, e.g., on an embedded device. However, with an abundance of compression techniques available it is not obvious how to proceed; many bring with them additional hyperparameter tuning, and are specific to particular network types. In this paper, we propose a simple compression technique that is general, easy to apply, and requires minimal tuning. Given a large, trained network, we propose (i) substituting its expensive convolutions with cheap alternatives, leaving the overall architecture unchanged; (ii) treating this new network as a student and training it with the original as a teacher through distillation. We demonstrate this approach separately for (i) networks predominantly consisting of full 3 ×3 convolutions and (ii) 1 ×1 or pointwise convolutions which together make up the vast majority of contemporary networks. We are able to leverage a number of methods that have been developed as efficient alternatives to fully-connected layers for pointwise substitution, allowing us provide Pareto-optimal benefits in efficiency/accuracy.
BibTeX
@article{crowley2021substituting,
  author = {Elliot J. Crowley and Gavin Gray and Jack Turner and Amos Storkey},
  title = {{Substituting Convolutions for Neural Network Compression}},
  journal = {IEEE Access},
  year = {2021},
  doi = {10.1109/access.2021.3086321},
}
Figure