Knowing when we do not know: Bayesian continual learning for sensing-based analysis tasks

Sandra Servia Rodríguez, Cecilia Mascolo, Young D. KwonarXiv 2021 · arXiv

Introduction

Deep learning models deployed in real-world sensing applications, such as mobile health monitoring or autonomous driving, face a significant challenge: catastrophic forgetting. When these models are trained on a sequence of new tasks, they tend to abruptly lose the knowledge acquired from previous ones. While existing research often focuses solely on minimizing this loss of memory, this paper argues for a more nuanced approach. In many practical scenarios, the priority might shift between accurately learning a critical new task and perfectly retaining an old one.

Furthermore, standard models are often "overconfident" even when incorrect. This research introduces a Bayesian framework for continual learning that allows for a tunable trade-off between "forgetting" and "intransigence" (the inability to learn new information) while enabling the model to quantify its own uncertainty—essentially "knowing when it does not know."

Methodology: $\beta$-Weighted Variational Continual Learning

The proposed framework builds upon Variational Continual Learning (VCL). VCL is a Bayesian method where the posterior distribution of model parameters from previous tasks serves as the "prior" for the current task. The authors' key innovation is the introduction of a $\beta$-weighted loss function.

In standard VCL, the loss function balances the likelihood of new data against the Kullback-Leibler (KL) divergence, which measures how much the model's parameters deviate from the established prior. By introducing a $\beta$ coefficient to the KL term, the model's behavior can be tuned:

  • High $\beta$: Penalizes deviations from previous knowledge more heavily, thereby prioritizing "remembering" and reducing forgetting.
  • Low $\beta$: Grants the model more flexibility to adapt to new data, prioritizing "learning" and reducing intransigence.

To enable uncertainty quantification, the model utilizes Monte Carlo sampling across multiple stochastic forward passes. Two primary metrics are derived: Predictive Entropy, which measures the information in the predictive distribution, and Mutual Information, which captures model confidence by assessing the dependence between predictions and the parameter posterior.

Experimental Evaluation

The framework was tested across two diverse sensing domains:

  1. Audio Analysis: Sequential tasks included Speaker Identification (SI), Ambient Scene Detection (ASD), Stress Detection (SD), and Emotion Recognition (ER).
  2. Activity Analysis: Tasks involved Human Activity Recognition (HAR) and Person Identification (PI) using accelerometer data.

Key Findings and Discussion

The Forgetting-Intransigence Trade-off

The experiments confirm that there is no "free lunch" in continual learning; gains in stability often come at the cost of plasticity. The $\beta$ parameter acts as a critical control knob for this balance.

Figure 1: Average forgetting across different task sequences for varying values of $\beta$. Higher $\beta$ values effectively minimize the loss of knowledge from previous tasks.

As seen in Figure 1, increasing $\beta$ significantly reduces the average forgetting across various task sequences. Conversely, Figure 2 illustrates the corresponding increase in intransigence when $\beta$ is high.

Figure 2: Intransigence levels for different task sequences. Lower $\beta$ values allow the model to better adapt to new, distinct tasks, reducing intransigence.

By tuning $\beta$, developers can tailor the model to specific application needs—for instance, prioritizing the retention of a core safety task (high $\beta$) or ensuring the model can quickly learn a new user's unique patterns (low $\beta$).

Performance and Uncertainty

The model achieved robust performance, with average accuracies of ~78% for audio tasks and ~73-80% for activity recognition. Crucially, the researchers found that predictive entropy was significantly higher for samples the model misclassified. This suggests that uncertainty estimates can serve as a reliable "safety filter," allowing systems to discard or flag unreliable inferences in real-time.

Mobile Deployment and Feasibility

To demonstrate real-world viability, the framework was deployed on a Google Pixel 2. Despite the overhead of Bayesian inference (requiring 100 forward passes for uncertainty estimation), the model remained efficient:

  • Memory: 5.1 MB (approximately double a standard point-estimate model).
  • Latency: 1.235 seconds per inference.
  • Power Consumption: ~0.0088% of battery per inference.

Conclusion and Future Directions

This work provides a practical path forward for deploying adaptive sensing systems. While the current "multi-head" architecture requires explicit task identity, the ability to manage the forgetting-intransigence trade-off and provide uncertainty estimates is a major step toward reliable edge AI. Future work will explore automated methods for determining task relatedness to dynamically adjust $\beta$ and strategies to reduce the computational cost of Bayesian sampling.

Summary updated

Abstract

Despite much research targeted at enabling conventional machine learning models to continually learn tasks and data distributions sequentially without forgetting the knowledge acquired, little effort has been devoted to account for more realistic situations where learning some tasks accurately might be more critical than forgetting previous ones. In this paper we propose a Bayesian inference based framework to continually learn a set of real-world, sensing-based analysis tasks that can be tuned to prioritize the remembering of previously learned tasks or the learning of new ones. Our experiments prove the robustness and reliability of the learned models to adapt to the changing sensing environment, and show the suitability of using uncertainty of the predictions to assess their reliability.
BibTeX
@misc{rodriguez2021knowing,
  author = {Sandra Servia Rodríguez and Cecilia Mascolo and Young D. Kwon},
  title = {{Knowing when we do not know: Bayesian continual learning for sensing-based analysis tasks}},
  year = {2021},
  eprint = {2106.05872},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.05872},
}
Figure