AutoShuffleNet: Learning Permutation Matrices via an Exact Lipschitz Continuous Penalty in Deep Convolutional Neural Networks

Jiancheng Lyu, Shuai Zhang, Yingyong Qi, Jack Xin

Introduction

Lightweight convolutional neural networks (LCNNs), such as the ShuffleNet family, are designed for efficient deployment on mobile and embedded devices. A cornerstone of these architectures is "channel shuffling," a hand-crafted operation that enables information flow between different feature groups. Traditionally, these shuffle patterns are fixed and manually designed. AutoShuffleNet proposes a shift from manual design to automated learning, replacing static shuffles with learned permutation matrices optimized directly from data.

Methodology: The $\ell_{1-2}$ Penalty

The primary challenge in learning permutations is their discrete nature, which makes them incompatible with standard gradient-based optimization like stochastic gradient descent (SGD). To bridge this gap, the authors introduce a novel, exact non-convex penalty based on the difference between $\ell_1$ and $\ell_2$ norms.

For a vector with non-negative elements that sum to 1, the difference between its $\ell_1$ norm and its $\ell_2$ norm is zero if and only if the vector is a "one-hot" vector (one element is 1, others are 0). By applying this $\ell_{1-2}$ penalty to every row and column of a matrix, the optimization process is incentivized to push the matrix toward a valid permutation matrix.

The behavior of the penalty function F across different values of the hyperparameter λ.

Optimization Algorithm

  1. Initialization: Shuffle matrices are initialized with random values.
  2. Training: Weights and shuffle matrices are updated simultaneously using SGD.
  3. Normalization: A Sinkhorn-style normalization is applied after each update to maintain the matrix as roughly "doubly stochastic" (rows and columns sum to 1).
  4. Rounding: Post-training, the relaxed matrices are rounded to discrete permutations for inference, ensuring zero additional computational overhead compared to standard ShuffleNet.

Architectural Integration

AutoShuffleNet integrates these learned permutations into existing ShuffleNet v1 and v2 blocks. By replacing the "Channel Shuffle" layer with an "Auto Shuffle" layer, the network can discover task-specific connectivity patterns that are often more complex and effective than human-designed patterns.

Comparison of standard ShuffleNet v1 and v2 blocks with their corresponding AutoShuffleNet counterparts.

Experimental Results

The authors evaluated AutoShuffleNet on CIFAR-10 and ImageNet, demonstrating consistent performance improvements over manual baselines:

  • CIFAR-10: ShuffleNet v1 (g=3) accuracy improved from 90.55% to 91.76%, while ShuffleNet v2 (1x) saw an increase from 91.90% to 92.81%.
  • ImageNet: The method yielded consistent top-1 accuracy gains ranging from 0.1% to 0.6%.
  • Stability: A key finding was the stability of the rounding process; the learned matrices were sufficiently close to true permutations that rounding caused negligible accuracy drops.

Conclusion

AutoShuffleNet demonstrates that the connectivity of lightweight networks can be effectively learned using a Lipschitz continuous penalty. This approach eliminates the need for manual shuffle design and provides a mathematically grounded framework for optimizing discrete architectural components within a continuous training regime. Future work may extend this methodology to other discrete neural architecture search (NAS) problems.

Summary updated

BibTeX
@inproceedings{lyu2020autoshufflenet,
  author = {Jiancheng Lyu and Shuai Zhang and Yingyong Qi and Jack Xin},
  title = {{AutoShuffleNet: Learning Permutation Matrices via an Exact Lipschitz Continuous Penalty in Deep Convolutional Neural Networks}},
  booktitle = {ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  year = {2020},
  url = {https://openreview.net/forum?id=X7wM4eksJv},
}
Figure