Practical Insights on Incremental Learning of New Human Physical Activity on the Edge

George Arvanitakis, Jingwei Zuo, Mthandazo Ndhlovu, Hakim Hacid

Abstract

Human Activity Recognition (HAR) is a cornerstone of modern wearable technology, yet traditional cloud-centric models struggle with privacy, latency, and the inability to adapt to new user behaviors. This paper introduces MAGNETO, a system designed for on-device incremental learning. By leveraging supervised contrastive learning and a compact support set, MAGNETO enables edge devices to learn novel physical activities with minimal computational overhead. Our results demonstrate that the system maintains high accuracy (above 93%) even after sequentially adding multiple new activities, while significantly reducing storage requirements without substantial performance loss.

Introduction

Standard HAR systems typically rely on static models trained on massive, centralized datasets. While effective for common activities like walking or running, these models are inherently inflexible. When a user engages in a new activity—such as riding an e-scooter—the model cannot adapt without transmitting raw sensor data to the cloud for retraining. This process raises significant privacy concerns, introduces network latency, and consumes substantial battery and bandwidth.

To address these challenges, we present MAGNETO, a framework for incremental learning on the edge. The goal is to allow a mobile device to "learn" new activities locally, ensuring data privacy and providing immediate adaptation to the user's lifestyle.

The MAGNETO Framework

The MAGNETO system operates in two distinct phases: Cloud Initialization and Edge Incremental Learning.

Cloud Initialization

The process begins with a five-layer fully connected neural network pre-trained using Supervised Contrastive Learning. Unlike traditional cross-entropy loss, which focuses solely on classification, contrastive learning optimizes the model to map similar activities into close proximity within a "latent space" while pushing dissimilar activities apart. This creates a robust feature extractor that is better suited for downstream adaptation.

Edge Incremental Learning

Once deployed, the device carries the pre-trained model and a Support Set ($D_s$), which consists of a small representative sample of the original training data. When a user wants to teach the device a new activity:

  1. Data Collection: The device captures a few samples of the new movement.
  2. Support Set Update: These new samples are integrated into the local Support Set.
  3. Local Fine-Tuning: The model is retrained (fine-tuned) locally using the updated Support Set.
  4. Prototype Update: The system calculates "Class Prototypes"—the average representation of an activity in the latent space—to include the new class.

Experimental Methodology

The system was evaluated using an extensive internal dataset comprising over 100GB of sensory data. The data includes five distinct activity classes: Drive, E-scooter, Run, Still, and Walk. Features were extracted from 22 mobile sensors (including accelerometers and gyroscopes) using one-second windows, resulting in 86 distinct features per window. To ensure the model's ability to generalize, a 10-second time buffer was maintained between training and test observations to prevent temporal data leakage.

Results and Discussion

Our evaluation focused on the feasibility of on-device learning under the strict resource constraints of mobile hardware.

The Necessity of Local Adaptation

Initial tests confirmed that "zero-shot" learning—attempting to recognize new classes without retraining—is insufficient. For instance, the model frequently confused "Running" with "Walking" before adaptation. However, local fine-tuning for as few as 10 epochs with only 200 samples allowed the model to clearly distinguish these activities in the latent space, demonstrating the power of contrastive pre-training.

Resource Efficiency and Convergence

A critical observation regarding resource optimization is the relationship between the support set size and model performance. We discovered that reducing the support set size by a factor of ten (from 5,000 to 500 samples) resulted in an accuracy degradation of less than 2%. This is a significant finding for devices with limited storage. Furthermore, the model demonstrated "agile convergence," typically reaching peak accuracy within 40 training batches, which minimizes the impact on battery life.

Sequential Learning Performance

MAGNETO was tested on its ability to learn multiple activities in sequence. The system started with an initial accuracy of 98.9% for two activities. As new activities were added, the accuracy remained remarkably stable:

  • Initial (2 activities): 98.9%
  • After 1st new activity: 96.0%
  • After 2nd new activity: 94.5%
  • After 3rd new activity: 93.1%

While a minor "catastrophic forgetting" effect was observed—where the addition of new classes slightly impacts the recognition of previous ones—the overall accuracy remained well above the threshold required for practical application.

Conclusion

MAGNETO demonstrates that incremental learning on the edge is not only possible but highly efficient. By combining contrastive learning with a small, manageable support set, we can provide users with personalized, private, and adaptive HAR systems. Future research will focus on mitigating sequential degradation over a larger number of classes and testing the framework on more complex or similar physical movements to find the breaking point of the edge-based approach.

Summary updated

BibTeX
@inproceedings{arvanitakis2023practical,
  author = {George Arvanitakis and Jingwei Zuo and Mthandazo Ndhlovu and Hakim Hacid},
  title = {{Practical Insights on Incremental Learning of New Human Physical Activity on the Edge}},
  booktitle = {DSAA},
  year = {2023},
  url = {https://openreview.net/forum?id=cd5tVqeZgT},
}
Figure