TL;DR
This paper introduces "mindful active learning" (EMMA), a novel framework for training activity recognition models from wearable sensor data. EMMA accounts for human limitations like query budget and memory retention, leading to more efficient and accurate learning, especially in challenging scenarios like mobile health monitoring for the elderly.
What Problem Does This Paper Address?
Training accurate machine learning models for activity recognition using wearable sensors requires a significant amount of labeled data. Collecting this data in real-world, uncontrolled environments is challenging and expensive. Active learning offers a solution by intelligently selecting which data points to label, but existing methods often overlook crucial human factors. Specifically, they assume an "oracle" (the human annotator) has unlimited capacity to respond to queries and perfect memory. This paper addresses the gap by acknowledging that humans have limited query budgets and their ability to recall past events degrades over time, especially with delays between an event and its annotation. This is particularly relevant for continuous health monitoring where the sheer volume of data can overwhelm users.
Methods
The core of this paper is the introduction of Mindful Active Learning and its computational framework, EMMA (Entropy-Memory Maximization). EMMA aims to maximize active learning performance by considering three factors:
- Informativeness of sensor data: How much a data point helps the model learn (measured by entropy).
- Query budget: The maximum number of queries the user can handle.
- Human memory: The user's ability to recall the correct label, modeled using the Ebbinghaus forgetting curve and incorporating memory strength and time lag.
The authors formulate this as an optimization problem to select a subset of observations that maximizes expected gain, defined as the product of informativeness and memory retention. They prove this problem is NP-hard and propose a greedy heuristic algorithm (Algorithm 1) to solve it efficiently. This algorithm iteratively selects the observation with the highest combined informativeness and memory retention score until the budget is exhausted.
Datasets
The authors evaluated EMMA on three publicly available, real-world datasets:
- HART: Collected from 30 human subjects performing six activities.
- DAS: Data from eight human subjects and 10 activity classes.
- AReM: Involved six activities and one human subject.
They did not introduce new benchmarks but used these existing datasets to simulate oracles with varying memory strengths and query budgets.
Key Insights and Ideas
- Human limitations are critical: Ignoring human cognitive constraints (memory, budget) can lead to suboptimal or even detrimental active learning performance, especially when memory is weak.
- Memory matters: Incorporating memory retention into active learning selection significantly improves accuracy, particularly when the query budget is small.
- Forgetting can hurt: When an oracle has weak memory, adding more labeled data can actually decrease accuracy if the added data includes incorrect labels due to forgetting.
- Context is key for mobile health: Mindful active learning is especially beneficial in human-centered mobile health settings and for elderly individuals with cognitive impairments.
Key Results
- EMMA achieved activity recognition accuracies ranging from 21% to 97%, depending on memory strength, query budget, and task difficulty.
- On average, EMMA achieved 13.5% higher accuracy than a method considering only data informativeness (EMA).
- EMMA's performance was at most 20% less than the experimental upper-bound (perfect oracle) and up to 80% higher than the experimental lower-bound (random selection).
- The performance improvement of EMMA over EMA and MMA was most notable with small budgets and weak memories.
- For weak memory cases, accuracy could decline after a certain point as more incorrect labels are introduced.
Limitations and Future Work
- Simulated Oracles: The study relied on simulated oracles. Future work includes conducting user studies with actual individuals, especially those with cognitive impairments, to validate findings quantitatively.
- Instantaneous Responses: The current model assumes the oracle responds to queries instantaneously. Future research will investigate context-sensitive active learning that accounts for delayed responses.
- Pool-based Active Learning: The current work focuses on pool-based active learning. The authors plan to develop mindful active learning strategies for on-the-fly query decisions as data becomes available in real-time.
Summary updated