Deep learning models deployed on mobile devices often need to adapt to new data generated in real-time. However, retraining from scratch is computationally expensive, and simple fine-tuning often leads to "catastrophic forgetting" of previous knowledge. This paper, published in IEEE Access, proposes a multi-step training framework that leverages selective sparsity training and adaptive distillation to efficiently incorporate new data while preserving existing model performance.
The Challenge: Data Evolution and Catastrophic Forgetting
In real-world applications, data is dynamic. Mobile devices constantly collect new information tailored to specific user environments. To keep Convolutional Neural Networks (CNNs) accurate, they must be updated. Two main obstacles prevent easy updates:
- Privacy and Scale: Original training data often remains on secure servers, and retraining large models on mobile hardware is impractical.
- Catastrophic Forgetting: Fine-tuning a model on new data alone causes it to lose the general knowledge acquired during initial training.
The Proposed Multi-Step Framework
The authors introduce a four-step iterative cycle designed to integrate new data batches without losing old knowledge.
1. Selective Sparsity Training (SST)
Unlike standard pruning that affects the entire network, SST adaptively targets a small percentage of "unimportant" filters. By forcing their scaling factors toward zero, the framework "empties" these slots for new information without degrading performance on the original dataset.
2. Diverse Re-initialization
The scaling factors of the emptied filters are reset to random values. Using different random ranges across training runs creates diversity, which is crucial for the subsequent distillation step.
3. Adaptive Online Distillation (AOD)
To learn from new data, the framework trains two "student" networks simultaneously. It employs an adaptive distillation loss where the better-performing student acts as a temporary teacher for the other, balancing the learning process and preventing the model from getting stuck in local minima.
4. Multi-step Loss (MSL)
In iterative scenarios, the model from the previous step serves as a frozen teacher. The MSL ensures the current student mimics the attention maps and outputs of its predecessor, effectively anchoring the new knowledge to the old.
Key Insights: Selective vs. Global Sparsity
A critical finding is the superiority of selective over global sparsity. Standard sparsity training (BST), often used for compression, eventually "kills" the network if applied repeatedly. SST is surgical, preserving the core knowledge base by only clearing what is currently unnecessary.
Experimental Results
The framework was evaluated on CIFAR-100, Tiny-ImageNet, and ImageNet across various architectures like ResNet and MobileNetv2.
- Superior Accuracy: The proposed method consistently outperformed standard fine-tuning. For ResNet34 on Tiny-ImageNet, it achieved 60.27% accuracy versus the baseline's 55.18%.
- Iterative Stability: The performance gap widened as more data steps were added. After four steps on CIFAR-100, the framework maintained 65.27% accuracy, while fine-tuning dropped to 60.66%.
Conclusion and Future Work
The multi-step framework provides a robust solution for updating CNNs on the fly. While it introduces some training overhead due to the dual-student AOD phase, its ability to prevent forgetting makes it highly effective for mobile and edge computing. Future research will explore more efficient re-initialization strategies and applications beyond image classification.
Summary updated