Multi-Step Training Framework Using Sparsity Training for Efficient Utilization of Accumulated New Data in Convolutional Neural Networks

Jeongjun Lee, Hyun Kim

Deep learning models deployed on mobile devices often need to adapt to new data generated in real-time. However, retraining from scratch is computationally expensive, and simple fine-tuning often leads to "catastrophic forgetting" of previous knowledge. This paper, published in IEEE Access, proposes a multi-step training framework that leverages selective sparsity training and adaptive distillation to efficiently incorporate new data while preserving existing model performance.

The Challenge: Data Evolution and Catastrophic Forgetting

In real-world applications, data is dynamic. Mobile devices constantly collect new information tailored to specific user environments. To keep Convolutional Neural Networks (CNNs) accurate, they must be updated. Two main obstacles prevent easy updates:

  1. Privacy and Scale: Original training data often remains on secure servers, and retraining large models on mobile hardware is impractical.
  2. Catastrophic Forgetting: Fine-tuning a model on new data alone causes it to lose the general knowledge acquired during initial training.

The Proposed Multi-Step Framework

The authors introduce a four-step iterative cycle designed to integrate new data batches without losing old knowledge.

1. Selective Sparsity Training (SST)

Unlike standard pruning that affects the entire network, SST adaptively targets a small percentage of "unimportant" filters. By forcing their scaling factors toward zero, the framework "empties" these slots for new information without degrading performance on the original dataset.

Selective Sparsity Training (SST) targets specific unimportant filters for removal to make room for new data.

2. Diverse Re-initialization

The scaling factors of the emptied filters are reset to random values. Using different random ranges across training runs creates diversity, which is crucial for the subsequent distillation step.

3. Adaptive Online Distillation (AOD)

To learn from new data, the framework trains two "student" networks simultaneously. It employs an adaptive distillation loss where the better-performing student acts as a temporary teacher for the other, balancing the learning process and preventing the model from getting stuck in local minima.

4. Multi-step Loss (MSL)

In iterative scenarios, the model from the previous step serves as a frozen teacher. The MSL ensures the current student mimics the attention maps and outputs of its predecessor, effectively anchoring the new knowledge to the old.

Key Insights: Selective vs. Global Sparsity

A critical finding is the superiority of selective over global sparsity. Standard sparsity training (BST), often used for compression, eventually "kills" the network if applied repeatedly. SST is surgical, preserving the core knowledge base by only clearing what is currently unnecessary.

Standard Global Sparsity Training (BST) can lead to excessive pruning of important filters, causing performance degradation over multiple steps.

Experimental Results

The framework was evaluated on CIFAR-100, Tiny-ImageNet, and ImageNet across various architectures like ResNet and MobileNetv2.

  • Superior Accuracy: The proposed method consistently outperformed standard fine-tuning. For ResNet34 on Tiny-ImageNet, it achieved 60.27% accuracy versus the baseline's 55.18%.
  • Iterative Stability: The performance gap widened as more data steps were added. After four steps on CIFAR-100, the framework maintained 65.27% accuracy, while fine-tuning dropped to 60.66%.

Conclusion and Future Work

The multi-step framework provides a robust solution for updating CNNs on the fly. While it introduces some training overhead due to the dual-student AOD phase, its ability to prevent forgetting makes it highly effective for mobile and edge computing. Future research will explore more efficient re-initialization strategies and applications beyond image classification.

Summary updated

Abstract

With the accumulation of large amounts of data that have the most significant impact on network performance and the development of hardware accelerators that can support such data processing, convolutional neural networks (CNNs) have achieved remarkable advances in various applications such as computer vision. In particular, in recent years, various new data suitable for user environments have been obtained from numerous mobile devices, and accordingly, methods have to be developed to train CNNs with only new data on the existing pre-trained network without access to the data used for the pre-training process on server platforms. Nevertheless, only a few studies have considered efficient training schemes for new data. In this study, we propose a multi-step training framework that efficiently utilizes accumulated new data for CNN training. In detail, to significantly improve the performance of CNNs, the proposed method creates unimportant filters while preserving knowledge of previous data through selective sparsity training and can effectively utilize these filters for CNN training of new data through diverse re-initialization and adaptive online distillation techniques. In addition, the proposed multi-step sparsity training with multi-step loss enables iterative network training of new data. The results of extensive experiments performed on various datasets show that the proposed method outperforms many existing methods, including fine-tuning.
BibTeX
@article{lee2023multi,
  author = {Jeongjun Lee and Hyun Kim},
  title = {{Multi-Step Training Framework Using Sparsity Training for Efficient Utilization of Accumulated New Data in Convolutional Neural Networks}},
  journal = {IEEE Access},
  year = {2023},
  doi = {10.1109/access.2023.3333902},
}
Figure