LCS: Learning Compressible Subspaces for Efficient, Adaptive, Real-Time Network Compression at Inference Time

Elvis Nunez, Maxwell Horton, Anish Prabhu, Anurag Ranjan, Ali Farhadi, Mohammad RastegariWACV 2023

TL;DR

This paper introduces LCS (Learning Compressible Subspaces), a novel method for training deep neural networks that can be compressed on-device in real-time to any compression level without retraining. This allows models to dynamically adapt to varying computational resources, offering a significant advantage for edge devices.

What Problem Does This Paper Address?

Deploying deep neural networks (DNNs) on resource-constrained devices like smartphones and watches presents a challenge: the available computational resources (CPU, memory, power) are not static. They can fluctuate due to other running processes or battery levels. Traditional DNNs are designed with a fixed architecture and resource usage, making them ill-suited for these dynamic environments. While training multiple models of different sizes or using techniques like Network Slimming and Universal Slimming can offer accuracy-efficiency trade-offs, they often require training and storing multiple models, pre-defining compression levels, or recalibrating BatchNorm statistics, which is computationally expensive and inflexible. This paper addresses the need for a single, adaptable model that can be compressed on-device in real-time to match available resources without significant overhead.

Methods

The core innovation is the concept of a "compressible subspace" of neural networks. Instead of training a single fixed network, LCS trains a model that spans a spectrum of potential network configurations.

  • Linear Subspace (LCS+L): This method trains a linear subspace defined by two sets of weights, $w_1$ and $w_2$. Any network within this subspace can be represented as $w^*(\alpha) = \alpha w_1 + (1 - \alpha) w_2$, where $\alpha$ is a parameter controlling the compression level. One endpoint ($\alpha \approx \alpha_1$) is optimized for high efficiency (high compression), while the other ($\alpha \approx \alpha_2$) is optimized for high accuracy (low compression). By varying $\alpha$, the model can achieve different accuracy-efficiency trade-offs.
  • Point Subspace (LCS+P): To eliminate parameter overhead, this approach trains a single set of weights, $w$. Compression is applied dynamically during inference using a compression function $f(w, \gamma(\alpha))$, where $\gamma(\alpha)$ determines the compression level based on $\alpha$. This method conditions a single network to tolerate varying levels of compression.
  • GroupNorm: To circumvent the need for BatchNorm recalibration (which is corrupted by compression), LCS replaces BatchNorm layers with GroupNorm. GroupNorm normalizes across groups of channels rather than across a batch, thus not requiring stored statistics that can become inaccurate.
  • Training Procedure: The training involves sampling a position $\alpha$ in the subspace, determining the compression level $\gamma(\alpha)$, applying compression using $f$, and then performing a standard forward and backward pass to update the weights (either $w_1, w_2$ for LCS+L or $w$ for LCS+P).

Datasets

The paper uses the following datasets:

  • CIFAR-10: Used for training and evaluating cPreResNet20.
  • ImageNet: Used for training and evaluating various CNN architectures (ResNet18, VGG19, MnasNet, MobileNetV2, MobileNetV3-Small, MobileNetV3-Large) and vision transformer models (DeiT-Ti, DeiT-S, CaiT-XXS).

Key Insights and Ideas

  • Adaptive Compression is Crucial: The paper strongly argues that static compression levels are insufficient for real-world dynamic environments. The ability to adapt compression at inference time is key.
  • Subspace Learning for Compression: The core idea of learning a "compressible subspace" is powerful. It allows a single trained model to represent a continuum of compressed versions, rather than discrete, pre-defined states.
  • GroupNorm as a Solution to BatchNorm Issues: Replacing BatchNorm with GroupNorm is a critical enabler for adaptive compression, as it removes the dependency on recalibrating statistics that are invalidated by compression.
  • LCS+P's Zero Overhead: The point subspace method (LCS+P) achieving compression without any additional parameter storage is a significant practical advantage for memory-constrained devices.
  • Generality of the Approach: The method is shown to be applicable to multiple compression techniques (unstructured sparsity, structured sparsity, and quantization), demonstrating its broad utility.

Key Results

  • Accuracy-Efficiency Trade-off: LCS models consistently achieve a strong accuracy-efficiency trade-off across various architectures and compression types.
  • Comparable to Baselines: At any given compression level, LCS models achieve accuracy comparable to baseline models specifically trained for that exact compression level.
  • Unstructured Sparsity: LCS+L+GN achieved better accuracy at high sparsities compared to networks trained for specific TopK sparsity targets, which degraded sharply at other sparsity levels.
  • Structured Sparsity: LCS models demonstrated a smoother accuracy-efficiency trade-off compared to "Discrete" and "Sandwich" baselines, which exhibited peaks and troughs. LCS+P+IN showed no parameter overhead.
  • Quantization: LCS models achieved high accuracy across various bit widths, matching or exceeding baselines trained for specific bit widths.
  • Runtime: Table 2 shows that LCS models offer significant reductions in runtime and memory with increasing sparsity, with minimal overhead for the LCS+P method.

Limitations and Future Work

  • Transformer Convergence for Structured Sparsity: Preliminary experiments with transformers for structured sparsity did not converge to high accuracies. The authors hypothesize this might be due to inter-channel variation in transformers.
  • Further Investigation on Transformers: More investigation is suggested for transformer models, particularly regarding the benefits of learning fewer parameters versus increased capacity.
  • Quantization Instabilities: The authors avoided lower bit widths in quantization experiments due to encountered training instabilities in baselines.
  • Future Work: The paper suggests further investigation into transformer models and exploring other compression techniques. The open-source code is provided for further research.

Summary updated

BibTeX
@inproceedings{nunez2023lcs,
  author = {Elvis Nunez and Maxwell Horton and Anish Prabhu and Anurag Ranjan and Ali Farhadi and Mohammad Rastegari},
  title = {{LCS: Learning Compressible Subspaces for Efficient, Adaptive, Real-Time Network Compression at Inference Time}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure