MobilePrune: Neural Network Compression via ℓ0 Sparse Group Lasso on the Mobile System

Yubo Shao, Kaikai Zhao, Zhiwen Cao, Zhehao Peng, Xingang Peng, Pan Li, Yijie Wang, Jianzhu MaSensors 2022 · OpenReview

TL;DR

This paper introduces MobilePrune, a novel neural network compression technique that uses ℓ0 sparse group lasso to achieve significant model size reduction and computational efficiency. By optimizing for both group-level and individual-level sparsity, MobilePrune enables deep learning models to run effectively on resource-constrained mobile devices, demonstrating reduced latency and battery consumption.

What Problem Does This Paper Address?

Deploying deep learning models on smartphones is challenging due to their substantial computational demands and memory footprints. While existing model compression techniques like weight pruning (ℓ1 or ℓ0 regularization) can reduce model size, they often result in unstructured sparsity. This unstructured sparsity, while reducing the number of non-zero weights, doesn't translate to practical speedups or memory savings in hardware accelerators like cuDNN, which rely on structured matrix operations (e.g., General Matrix Multiply - GEMM). Structured pruning methods aim to address this by pruning entire groups of weights, but they may not fully exploit the potential for sparsity. The authors aim to bridge this gap by developing a method that achieves both structured sparsity beneficial for hardware acceleration and fine-grained sparsity for further optimization, all while ensuring compatibility with existing mobile platforms and hardware.

Methods

The core of MobilePrune is its novel ℓ0 sparse group lasso regularizer. This approach combines two types of sparsity:

  • Group Lasso: This enforces sparsity at the group level. For fully connected layers, it prunes entire neurons (and their associated weights). For convolutional layers, it prunes entire filters or channels, which directly benefits GEMM operations. This structured sparsity is crucial for hardware acceleration.
  • ℓ0 Norm: This enforces sparsity at the individual weight level within the remaining groups. This allows for further fine-tuning and optimization of the model.

The authors developed a new optimization algorithm, MobilePrune, based on the Proximal Alternating Linearized Minimization (PALM) framework. A key contribution is the development of efficient and exact algorithms to solve the proximal operators for the ℓ0 sparse group lasso, which are novel and have not been previously addressed. They also allow for complex group structures, such as overlapping groups and tree structures, to accommodate diverse DNN architectures.

Datasets

The paper evaluates MobilePrune on several standard image and activity recognition datasets:

  • Image Benchmarks:
    • MNIST
    • CIFAR-10
    • Tiny-ImageNet
  • Human Activity Recognition (HAR) Benchmarks:
    • WISDM
    • UCI-HAR
    • PAMAP2

The authors also mention that the source codes for the proposed model and a mobile application are available, suggesting a potential for new benchmarks or evaluation platforms.

Key Insights and Ideas

  • Synergy of Structured and Unstructured Sparsity: The paper highlights that combining structured sparsity (group lasso) with fine-grained sparsity (ℓ0 norm) is more effective than either alone. Group lasso makes models compatible with hardware accelerators, while ℓ0 norm provides further compression.
  • GEMM Compatibility is Key: The authors emphasize that modern deep learning heavily relies on GEMM operations. Their structured pruning strategy is designed to directly benefit these operations, leading to practical speedups.
  • Exact ℓ0 Optimization: Developing an algorithm that can optimize the ℓ0 norm exactly and guarantee global convergence is a significant theoretical contribution, as many existing methods rely on relaxations or have unproven convergence.
  • Mobile Deployment Matters: The authors go beyond theoretical and desktop-based evaluations by deploying their compressed models on actual Android devices, providing real-world evidence of reduced latency and battery consumption.

Key Results

  • Significant Compression: MobilePrune achieves substantial reductions in model size (parameters) and computational costs (FLOPs) across various popular deep learning models and datasets. For example, on LeNet-300-100, they achieved a parameter reduction to 5.25 million with only a 0.01% accuracy drop.
  • Improved Mobile Performance: On Android devices, MobilePrune demonstrated significant savings in response delay (e.g., up to 61.94% on WISDM) and battery consumption (e.g., up to 37.50% on WISDM) compared to other pruning methods.
  • State-of-the-Art Performance: MobilePrune consistently outperformed other state-of-the-art structured pruning methods in terms of accuracy, compression ratio, and efficiency on multiple benchmarks.
  • Hardware Acceleration Potential: The structured sparsity induced by MobilePrune is designed to be compatible with hardware accelerators like systolic tensor arrays, promising further performance gains.

Limitations and Future Work

  • Computational Cost of Optimization: While the resulting models are efficient, the optimization process for the ℓ0 sparse group lasso can be computationally intensive. The paper mentions that the computational burden mainly concentrates on partial derivative calculations, similar to training a normal DNN.
  • Hardware Compatibility: While the paper discusses compatibility with cuDNN and systolic tensor arrays, the full potential of the induced sparsity on specific hardware accelerators would require further investigation.
  • Exploration of Other Architectures: The authors suggest that MobilePrune could be adopted on other mobile-friendly architectures like Neural Logic Circuits and Binary Neural Networks, which could be a direction for future work. They also plan to explore which mobile neural network architectures are best suited for integration with their compression model.

Summary updated

Abstract

It is hard to directly deploy deep learning models on today’s smartphones due to the substantial computational costs introduced by millions of parameters. To compress the model, we develop an ℓ0-based sparse group lasso model called MobilePrune which can generate extremely compact neural network models for both desktop and mobile platforms. We adopt group lasso penalty to enforce sparsity at the group level to benefit General Matrix Multiply (GEMM) and develop the very first algorithm that can optimize the ℓ0 norm in an exact manner and achieve the global convergence guarantee in the deep learning context. MobilePrune also allows complicated group structures to be applied on the group penalty (i.e., trees and overlapping groups) to suit DNN models with more complex architectures. Empirically, we observe the substantial reduction of compression ratio and computational costs for various popular deep learning models on multiple benchmark datasets compared to the state-of-the-art methods. More importantly, the compression models are deployed on the android system to confirm that our approach is able to achieve less response delay and battery consumption on mobile phones.
BibTeX
@inproceedings{shao2022mobileprune,
  author = {Yubo Shao and Kaikai Zhao and Zhiwen Cao and Zhehao Peng and Xingang Peng and Pan Li and Yijie Wang and Jianzhu Ma},
  title = {{MobilePrune: Neural Network Compression via ℓ0 Sparse Group Lasso on the Mobile System}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=mHojIWgvUO},
}
Figure