TL;DR
This paper introduces LCS (Learning Compressible Subspaces), a novel method for training deep neural networks that can be compressed on-device in real-time to any compression level without retraining. This allows models to dynamically adapt to varying computational resources, offering a significant advantage for edge devices.
What Problem Does This Paper Address?
Deploying deep neural networks (DNNs) on resource-constrained devices like smartphones and watches presents a challenge: the available computational resources (CPU, memory, power) are not static. They can fluctuate due to other running processes or battery levels. Traditional DNNs are designed with a fixed architecture and resource usage, making them ill-suited for these dynamic environments. While training multiple models of different sizes or using techniques like Network Slimming and Universal Slimming can offer accuracy-efficiency trade-offs, they often require training and storing multiple models, pre-defining compression levels, or recalibrating BatchNorm statistics, which is computationally expensive and inflexible. This paper addresses the need for a single, adaptable model that can be compressed on-device in real-time to match available resources without significant overhead.
Methods
The core innovation is the concept of a "compressible subspace" of neural networks. Instead of training a single fixed network, LCS trains a model that spans a spectrum of potential network configurations.
- Linear Subspace (LCS+L): This method trains a linear subspace defined by two sets of weights, $w_1$ and $w_2$. Any network within this subspace can be represented as $w^*(\alpha) = \alpha w_1 + (1 - \alpha) w_2$, where $\alpha$ is a parameter controlling the compression level. One endpoint ($\alpha \approx \alpha_1$) is optimized for high efficiency (high compression), while the other ($\alpha \approx \alpha_2$) is optimized for high accuracy (low compression). By varying $\alpha$, the model can achieve different accuracy-efficiency trade-offs.
- Point Subspace (LCS+P): To eliminate parameter overhead, this approach trains a single set of weights, $w$. Compression is applied dynamically during inference using a compression function $f(w, \gamma(\alpha))$, where $\gamma(\alpha)$ determines the compression level based on $\alpha$. This method conditions a single network to tolerate varying levels of compression.
- GroupNorm: To circumvent the need for BatchNorm recalibration (which is corrupted by compression), LCS replaces BatchNorm layers with GroupNorm. GroupNorm normalizes across groups of channels rather than across a batch, thus not requiring stored statistics that can become inaccurate.
- Training Procedure: The training involves sampling a position $\alpha$ in the subspace, determining the compression level $\gamma(\alpha)$, applying compression using $f$, and then performing a standard forward and backward pass to update the weights (either $w_1, w_2$ for LCS+L or $w$ for LCS+P).
Datasets
The paper uses the following datasets:
- CIFAR-10: Used for training and evaluating cPreResNet20.
- ImageNet: Used for training and evaluating various CNN architectures (ResNet18, VGG19, MnasNet, MobileNetV2, MobileNetV3-Small, MobileNetV3-Large) and vision transformer models (DeiT-Ti, DeiT-S, CaiT-XXS).
Key Insights and Ideas
- Adaptive Compression is Crucial: The paper strongly argues that static compression levels are insufficient for real-world dynamic environments. The ability to adapt compression at inference time is key.
- Subspace Learning for Compression: The core idea of learning a "compressible subspace" is powerful. It allows a single trained model to represent a continuum of compressed versions, rather than discrete, pre-defined states.
- GroupNorm as a Solution to BatchNorm Issues: Replacing BatchNorm with GroupNorm is a critical enabler for adaptive compression, as it removes the dependency on recalibrating statistics that are invalidated by compression.
- LCS+P's Zero Overhead: The point subspace method (LCS+P) achieving compression without any additional parameter storage is a significant practical advantage for memory-constrained devices.
- Generality of the Approach: The method is shown to be applicable to multiple compression techniques (unstructured sparsity, structured sparsity, and quantization), demonstrating its broad utility.
Key Results
- Accuracy-Efficiency Trade-off: LCS models consistently achieve a strong accuracy-efficiency trade-off across various architectures and compression types.
- Comparable to Baselines: At any given compression level, LCS models achieve accuracy comparable to baseline models specifically trained for that exact compression level.
- Unstructured Sparsity: LCS+L+GN achieved better accuracy at high sparsities compared to networks trained for specific TopK sparsity targets, which degraded sharply at other sparsity levels.
- Structured Sparsity: LCS models demonstrated a smoother accuracy-efficiency trade-off compared to "Discrete" and "Sandwich" baselines, which exhibited peaks and troughs. LCS+P+IN showed no parameter overhead.
- Quantization: LCS models achieved high accuracy across various bit widths, matching or exceeding baselines trained for specific bit widths.
- Runtime: Table 2 shows that LCS models offer significant reductions in runtime and memory with increasing sparsity, with minimal overhead for the LCS+P method.
Limitations and Future Work
- Transformer Convergence for Structured Sparsity: Preliminary experiments with transformers for structured sparsity did not converge to high accuracies. The authors hypothesize this might be due to inter-channel variation in transformers.
- Further Investigation on Transformers: More investigation is suggested for transformer models, particularly regarding the benefits of learning fewer parameters versus increased capacity.
- Quantization Instabilities: The authors avoided lower bit widths in quantization experiments due to encountered training instabilities in baselines.
- Future Work: The paper suggests further investigation into transformer models and exploring other compression techniques. The open-source code is provided for further research.
Summary updated