Diagonalwise Refactorization: An Efficient Training Method for Depthwise Convolutions

Zheng Qin, Zhaoning Zhang, Dongsheng Li, Yiming Zhang, Yuxing Peng

Abstract

Depthwise convolutions significantly reduce parameter counts and theoretical computations in neural networks, yet they often suffer from poor training performance on GPUs. This paper introduces "diagonalwise refactorization," a mathematical technique that rearranges depthwise weights into a large diagonal matrix. This transformation allows depthwise operations to be processed as standard convolutions, enabling the use of highly optimized libraries such as NVIDIA's cuDNN. Experimental results demonstrate training speedups of up to 15.4x across major deep learning frameworks with negligible memory overhead.

Introduction

Lightweight convolutional neural networks (ConvNets), such as MobileNet and Xception, rely heavily on depthwise separable convolutions to achieve efficiency. Unlike standard convolutions that operate across all input channels simultaneously, a depthwise separable convolution splits the process into two stages: a depthwise convolution followed by a pointwise ($1 \times 1$) convolution.

Structure of a depthwise separable convolution, consisting of depthwise and pointwise stages.

While this structure drastically reduces the number of multiply-add operations and parameters, it introduces a significant bottleneck during training. On parallel hardware like GPUs, depthwise convolutions exhibit a low computation-to-memory-access ratio. Current implementations typically use either inefficient channel-by-channel kernel launches or specialized CUDA kernels that cannot leverage the deep optimizations found in libraries like cuDNN. Consequently, depthwise layers can account for over 80% of training time despite representing only a small fraction of total FLOPs.

Methodology: Diagonalwise Refactorization

The authors propose Diagonalwise Refactorization to bridge the gap between theoretical efficiency and hardware performance. The core idea is to mathematically reframe a depthwise convolution so it can be executed as a standard convolution.

Matrix Rearrangement and Masking

Instead of maintaining $M$ independent $K \times K$ filters, the method constructs a single large weight matrix. The individual channel filters are placed along the diagonal of this new matrix, with all other entries set to zero. To maintain the mathematical properties of a depthwise convolution during training, a constant binary mask is applied. This mask ensures that only the diagonal elements are updated during backpropagation, preventing the network from learning cross-channel dependencies that should not exist in a depthwise layer.

The proposed diagonalwise refactorization method, converting depthwise convolution into a standard convolution format.

Optimization and Scaling

By refactoring the operation into a standard convolution, the framework can utilize NVIDIA's cuDNN library, which includes highly tuned microarchitecture-level optimizations like Winograd transforms. To handle layers with a very large number of channels (e.g., 1024), where a single diagonal matrix would be excessively sparse and memory-intensive, the authors implement a grouping mechanism. Channels are divided into smaller blocks (e.g., 32 channels per group), and refactorization is applied independently to each group, balancing compute efficiency and memory usage.

Experimental Evaluation

The performance of diagonalwise refactorization was evaluated based on hardware throughput rather than traditional image classification metrics, as the contribution is purely computational. Benchmarks were conducted using MobileNet architectures on an NVIDIA GTX 1080Ti across five major deep learning frameworks.

Key Results

The method yielded substantial training speedups compared to native depthwise implementations:

  • Darknet: 15.4x speedup
  • Caffe: 8.4x speedup
  • PyTorch: 5.4x speedup
  • MXNet: 3.5x speedup
  • TensorFlow: 1.4x speedup

The speedup in TensorFlow is less dramatic because it already utilizes specialized kernels, whereas frameworks like Caffe and PyTorch previously relied on slower channel-by-channel approaches. Notably, the memory overhead was found to be negligible; for instance, in Caffe, memory consumption increased by less than 0.5% (from 7947 MB to 7981 MB).

Discussion and Conclusion

A primary insight of this work is that theoretical FLOPs are often a poor predictor of actual hardware performance. Memory bandwidth, kernel launch overhead, and the availability of optimized libraries like cuDNN play a more decisive role in training speed. By "freeloading" on these existing optimizations through clever mathematical reframing, researchers can achieve massive performance gains without writing complex custom CUDA code.

Furthermore, the authors identified that the backward pass is a major bottleneck for specialized depthwise kernels due to atomic operations and memory write hazards. The cuDNN-backed approach avoids these issues entirely. Future work may involve extending this masking and refactorization strategy to other sparse operations, such as network pruning or generalized group convolutions in architectures like ShuffleNet.

While the method introduces some redundant zero-multiplications, these are effectively hidden by the massive parallelism of modern GPUs. Diagonalwise refactorization thus provides a practical and highly effective solution for accelerating the training of modern lightweight neural networks.

Summary updated

BibTeX
@inproceedings{qin2018diagonalwise,
  author = {Zheng Qin and Zhaoning Zhang and Dongsheng Li and Yiming Zhang and Yuxing Peng},
  title = {{Diagonalwise Refactorization: An Efficient Training Method for Depthwise Convolutions}},
  booktitle = {IJCNN},
  year = {2018},
  url = {https://openreview.net/forum?id=WyvqfxmExv},
}
Figure