Exploring System Performance of Continual Learning for Mobile and Embedded Sensing Applications

Young D. Kwon, Jagmohan Chauhan, Abhishek Kumar, Pan Hui, Cecilia Mascolo

Abstract

Deep learning models on mobile devices often remain static, failing to adapt to dynamic real-world environments. This paper presents the first comprehensive empirical study of Continual Learning (CL) specifically for mobile and embedded sensor data. By evaluating various incremental learning methods on actual mobile hardware, the study identifies the iCaRL algorithm as a highly effective solution that balances accuracy and system overhead, enabling privacy-preserving on-device adaptation.

Introduction

While deep learning has revolutionized mobile applications, deployed models typically lack the ability to learn from new data post-deployment. For instance, a user might take up a new physical activity that the original model cannot recognize. Continual Learning (CL) aims to address this by allowing models to learn new tasks or classes over time. However, the primary challenge is Catastrophic Forgetting (CF), where new training causes the model to lose previously learned knowledge.

Unlike Computer Vision, where CL is well-studied, sensor-based time-series data presents unique challenges. Furthermore, the resource constraints of mobile devices—CPU, memory, and battery—are often overlooked. This research bridges these gaps by evaluating CL methods on mobile hardware using sensing-specific datasets.

Methodology and Framework

The authors developed an end-to-end CL framework to evaluate six state-of-the-art Incremental Learning (IL) methods across three paradigms:

  1. Regularization (EWC, Online EWC, SI): Penalizes weight changes to preserve important knowledge.
  2. Replay (LwF): Uses knowledge distillation from previous model versions.
  3. Replay with Exemplars (iCaRL, GEM): Stores a small subset of raw data from old tasks to rehearse during new training.
Overview of the incremental learning workflow, showing the transition from a base model to an incremental model as new tasks are added.

The evaluation focused on three learning scenarios, with the most realistic being the sequential addition of new classes to a base classifier. Testing was performed on an Nvidia Jetson Nano (GPU) and a OnePlus 7 Pro (CPU) using datasets from Human Activity Recognition (HAR), Gesture Recognition (GR), and Emotion Recognition (ER).

Key Insights and Performance Analysis

The study reveals several critical findings regarding the feasibility of on-device CL:

The Superiority of Exemplars

In complex class-incremental scenarios, iCaRL consistently outperformed other methods. Its robustness stems from using a "Nearest-Class-Mean" (NCM) classification rule, which handles changes in data representation better than standard softmax layers.

Performance comparison (Weighted F1-score) of various CL methods on Ninapro (LOUO) and EmotionSense datasets, highlighting the stability of iCaRL compared to others.

System Feasibility and Privacy

  • Accuracy: iCaRL maintained F1-scores within 10% of a "Joint Model" (the theoretical upper bound). Regularization methods often suffered performance drops of 40-60% in complex scenarios.
  • On-Device Training: Training times on a smartphone CPU (20-75 minutes) align well with a "train-while-charging" strategy, typically performed overnight.
  • Storage and Privacy: Storing exemplars requires minimal space (under 115 MB), making it practical for modern devices. Crucially, local training keeps sensitive sensor data on the device, enhancing user privacy.

Conclusion and Future Directions

This research demonstrates that on-device incremental learning is both feasible and practical for mobile sensing. While iCaRL provides an excellent trade-off, future work should explore even more resource-constrained environments like microcontrollers. Additionally, investigating optimization techniques like quantization and the application of CL to newer architectures like Transformers remains a promising avenue for enhancing mobile intelligence.

Summary updated

BibTeX
@inproceedings{kwon2021exploring,
  author = {Young D. Kwon and Jagmohan Chauhan and Abhishek Kumar and Pan Hui and Cecilia Mascolo},
  title = {{Exploring System Performance of Continual Learning for Mobile and Embedded Sensing Applications}},
  booktitle = {SEC},
  year = {2021},
  doi = {10.1145/3453142.3491285},
}
Figure