Introduction
Deep learning models deployed in real-world sensing applications, such as mobile health monitoring or autonomous driving, face a significant challenge: catastrophic forgetting. When these models are trained on a sequence of new tasks, they tend to abruptly lose the knowledge acquired from previous ones. While existing research often focuses solely on minimizing this loss of memory, this paper argues for a more nuanced approach. In many practical scenarios, the priority might shift between accurately learning a critical new task and perfectly retaining an old one.
Furthermore, standard models are often "overconfident" even when incorrect. This research introduces a Bayesian framework for continual learning that allows for a tunable trade-off between "forgetting" and "intransigence" (the inability to learn new information) while enabling the model to quantify its own uncertainty—essentially "knowing when it does not know."
Methodology: $\beta$-Weighted Variational Continual Learning
The proposed framework builds upon Variational Continual Learning (VCL). VCL is a Bayesian method where the posterior distribution of model parameters from previous tasks serves as the "prior" for the current task. The authors' key innovation is the introduction of a $\beta$-weighted loss function.
In standard VCL, the loss function balances the likelihood of new data against the Kullback-Leibler (KL) divergence, which measures how much the model's parameters deviate from the established prior. By introducing a $\beta$ coefficient to the KL term, the model's behavior can be tuned:
- High $\beta$: Penalizes deviations from previous knowledge more heavily, thereby prioritizing "remembering" and reducing forgetting.
- Low $\beta$: Grants the model more flexibility to adapt to new data, prioritizing "learning" and reducing intransigence.
To enable uncertainty quantification, the model utilizes Monte Carlo sampling across multiple stochastic forward passes. Two primary metrics are derived: Predictive Entropy, which measures the information in the predictive distribution, and Mutual Information, which captures model confidence by assessing the dependence between predictions and the parameter posterior.
Experimental Evaluation
The framework was tested across two diverse sensing domains:
- Audio Analysis: Sequential tasks included Speaker Identification (SI), Ambient Scene Detection (ASD), Stress Detection (SD), and Emotion Recognition (ER).
- Activity Analysis: Tasks involved Human Activity Recognition (HAR) and Person Identification (PI) using accelerometer data.
Key Findings and Discussion
The Forgetting-Intransigence Trade-off
The experiments confirm that there is no "free lunch" in continual learning; gains in stability often come at the cost of plasticity. The $\beta$ parameter acts as a critical control knob for this balance.
As seen in Figure 1, increasing $\beta$ significantly reduces the average forgetting across various task sequences. Conversely, Figure 2 illustrates the corresponding increase in intransigence when $\beta$ is high.
By tuning $\beta$, developers can tailor the model to specific application needs—for instance, prioritizing the retention of a core safety task (high $\beta$) or ensuring the model can quickly learn a new user's unique patterns (low $\beta$).
Performance and Uncertainty
The model achieved robust performance, with average accuracies of ~78% for audio tasks and ~73-80% for activity recognition. Crucially, the researchers found that predictive entropy was significantly higher for samples the model misclassified. This suggests that uncertainty estimates can serve as a reliable "safety filter," allowing systems to discard or flag unreliable inferences in real-time.
Mobile Deployment and Feasibility
To demonstrate real-world viability, the framework was deployed on a Google Pixel 2. Despite the overhead of Bayesian inference (requiring 100 forward passes for uncertainty estimation), the model remained efficient:
- Memory: 5.1 MB (approximately double a standard point-estimate model).
- Latency: 1.235 seconds per inference.
- Power Consumption: ~0.0088% of battery per inference.
Conclusion and Future Directions
This work provides a practical path forward for deploying adaptive sensing systems. While the current "multi-head" architecture requires explicit task identity, the ability to manage the forgetting-intransigence trade-off and provide uncertainty estimates is a major step toward reliable edge AI. Future work will explore automated methods for determining task relatedness to dynamically adjust $\beta$ and strategies to reduce the computational cost of Bayesian sampling.
Summary updated