Building a Compact Convolutional Neural Network for Embedded Intelligent Sensor Systems Using Group Sparsity and Knowledge Distillation

Jungchan Cho, Minsik Lee

TL;DR

This paper tackles the challenge of deploying deep learning models on resource-constrained embedded sensors by proposing a novel two-pronged approach. They combine group sparsity regularization with knowledge distillation to automatically create compact networks, and introduce a feedback control mechanism to dynamically balance accuracy and model size during training, leading to improved trade-off curves.

What Problem Does This Paper Address?

The increasing prevalence of AI and deep learning (DL) has spurred a desire to embed these powerful capabilities into everyday devices, particularly intelligent sensors. However, a significant hurdle is the inherent trade-off between a deep neural network's complexity (number of parameters) and its computational requirements (processing time). Large, complex models, while accurate, are often too demanding for the limited memory, processing power, and energy budgets of embedded systems. Existing methods for creating smaller networks, such as knowledge distillation or parameter reduction (e.g., through sparsity), often struggle to maintain accuracy or require manual tuning of network architecture. This paper aims to bridge this gap by developing a method that automatically designs compact networks while preserving high performance, making DL more feasible for intelligent sensor applications.

Methods

The authors propose two key strategies:

  1. Group Sparsity and Knowledge Distillation (KD):
    • Group Sparsity: Instead of standard L1 regularization which can lead to irregular memory access patterns, they employ filter-wise group sparsity. This approach penalizes entire filters (groups of weights) to be zeroed out. This directly reduces the number of channels in convolutional layers, leading to a more structured and computationally efficient network. The advantage is that it automatically determines the number of parameters without manual architectural design.
  2. Knowledge Distillation (KD): To mitigate potential accuracy loss from sparsity, they leverage KD. A larger, pre-trained "teacher" network guides a smaller "student" network during training. The student learns not only from the ground truth labels but also from the softened outputs of the teacher, effectively transferring knowledge and compensating for reduced capacity.
  3. Proportional Feedback Control for Regularization:
    • The Balancing Act: A challenge with the first strategy is finding the optimal balance between the accuracy gained from KD and the parameter reduction from sparsity. This is where the second strategy comes in.
  4. Control Mechanism: They introduce a feedback control loop based on proportional control theory. This loop dynamically adjusts the regularization parameter ($\lambda_\gamma$) that controls the strength of the sparsity penalty.
  5. Error Signal: The "error signal" for the controller is derived from the difference in loss values between the teacher and student models. Specifically, it uses a weighted difference of the cross-entropy losses, where a hyperparameter $\gamma$ (relaxation variable) allows the student's target performance to be slightly less than the teacher's.
  6. Dynamic Adjustment: If the student's performance lags behind the teacher, the controller reduces the sparsity emphasis to prioritize accuracy. Conversely, if the student performs well, the controller can increase sparsity. This adaptive approach aims to find an optimal trade-off point and even improve the overall trade-off curve.

Datasets

The proposed method was evaluated on three standard image classification datasets:

  • CIFAR-10: A dataset with 10 classes and 32x32 resolution images.
  • CIFAR-100: A dataset with 100 classes and 32x32 resolution images.
  • ImageNet32x32: A downsampled version of ImageNet, with 1000 classes and 32x32 resolution images.

These datasets are commonly used for evaluating image recognition models and are suitable for testing efficiency gains on smaller-scale tasks relevant to embedded systems.

Key Insights and Ideas

  • Filter-wise Group Sparsity is Key: Zeroing out entire filters is more effective for structured model compression and hardware acceleration than individual weight pruning. It directly translates to fewer computations and memory accesses.
  • KD as a Sparsity Compensator: Knowledge distillation is not just for model compression; it's a powerful tool to recover accuracy lost due to aggressive sparsity regularization.
  • Feedback Control for Optimal Trade-offs: The most significant insight is that a dynamic feedback control mechanism can intelligently navigate the accuracy-sparsity trade-off. Instead of a static regularization parameter, an adaptive approach leads to better overall performance curves.
  • Improved Trade-off Curves: The proposed method doesn't just find a good point on the trade-off curve; it appears to improve the curve itself, offering better accuracy for a given level of sparsity than existing methods.
  • Automatic Architecture Design: The combination of group sparsity and KD allows for automatic determination of network parameters, reducing the need for manual architecture search.

Key Results

The paper demonstrates the effectiveness of their proposed method across the tested datasets:

  • Superior Performance: The proposed method consistently achieved better accuracy than baseline methods that used either sparsity alone or sparsity with KD but without the feedback control.
  • Stable Results: The feedback control mechanism led to more stable performance, especially when dealing with varying sparsity levels, compared to methods without control.
  • CIFAR-10/100: For these datasets, the proposed approach showed significant improvements, particularly as sparsity increased.
  • ImageNet32x32: On this larger dataset, KD-based methods initially showed slightly worse performance than non-KD methods at near-zero sparsity. However, by adjusting the $\gamma$ parameter to prioritize accuracy, the proposed method improved over baselines.
  • Hyperparameter $\gamma$ Analysis: The study showed that $\gamma=0.8$ was a good choice for balancing accuracy and sparsity, suggesting that a slightly relaxed accuracy target for the student can be beneficial.

Limitations and Future Work

  • Hyperparameter Tuning: While the feedback control aims to automate regularization, the hyperparameter $\gamma$ still requires tuning. The paper suggests $\gamma=0.8$ as a good starting point, but its optimal value might vary across tasks.
  • Computational Cost of Teacher: The method relies on a pre-trained teacher model. The efficiency gains are realized in the student, but the initial training of the teacher can still be computationally intensive.
  • Future Work: The authors suggest that their strategies can make deep networks more practical for intelligent sensors in resource-constrained platforms. Further exploration into different network architectures, more complex tasks, and real-world embedded system deployment would be valuable. The adaptive nature of the control mechanism also opens doors for more sophisticated control strategies.

Summary updated

Abstract

As artificial intelligence (AI)- or deep-learning-based technologies become more popular, the main research interest in the field is not only on their accuracy, but also their efficiency, e.g., the ability to give immediate results on the users’ inputs. To achieve this, there have been many attempts to embed deep learning technology on intelligent sensors. However, there are still many obstacles in embedding a deep network in sensors with limited resources. Most importantly, there is an apparent trade-off between the complexity of a network and its processing time, and finding a structure with a better trade-off curve is vital for successful applications in intelligent sensors. In this paper, we propose two strategies for designing a compact deep network that maintains the required level of performance even after minimizing the computations. The first strategy is to automatically determine the number of parameters of a network by utilizing group sparsity and knowledge distillation (KD) in the training process. By doing so, KD can compensate for the possible losses in accuracy caused by enforcing sparsity. Nevertheless, a problem in applying the first strategy is the unclarity in determining the balance between the accuracy improvement due to KD and the parameter reduction by sparse regularization. To handle this balancing problem, we propose a second strategy: a feedback control mechanism based on the proportional control theory. The feedback control logic determines the amount of emphasis to be put on network sparsity during training and is controlled based on the comparative accuracy losses of the teacher and student models in the training. A surprising fact here is that this control scheme not only determines an appropriate trade-off point, but also improves the trade-off curve itself. The results of experiments on CIFAR-10, CIFAR-100, and ImageNet32 × 32 datasets show that the proposed method is effective in building a compact network while preventing performance degradation due to sparsity regularization much better than other baselines.
BibTeX
@inproceedings{cho2019building,
  author = {Jungchan Cho and Minsik Lee},
  title = {{Building a Compact Convolutional Neural Network for Embedded Intelligent Sensor Systems Using Group Sparsity and Knowledge Distillation}},
  booktitle = {Sensors},
  year = {2019},
  url = {https://openreview.net/forum?id=eKUk01iZ8H},
}
Figure