TL;DR
This paper introduces "Skeptical Learning" (SKEL), a novel interactive machine learning framework designed to improve personal context recognition accuracy, especially when labels are provided by non-expert users. SKEL tackles the issue of noisy user-generated labels by enabling the machine to actively question and refine both its own predictions and the user's input, leading to more robust and accurate context awareness.
What Problem Does This Paper Address?
In many applications like pervasive and ubiquitous computing, recognizing a user's context (e.g., "What are you doing?", "Where are you?", "Who are you with?") is crucial for delivering personalized services. Supervised learning is commonly used for this, but its performance heavily relies on the quality of the training data's labels. The challenge arises when these labels are provided by the end-users themselves, who are often not domain experts and can introduce a significant amount of noise due to various biases (memory, unwillingness to report, carelessness). Traditional machine learning algorithms can only tolerate a small fraction of such noisy labels, leading to decreased prediction performance and increased complexity. This paper addresses the critical gap of handling unreliable, non-expert annotations in personal context recognition systems.
Methods
The core contribution is the Skeptical Learning (SKEL) framework, an interactive machine learning approach. SKEL's architecture, depicted in Figure 1, features a multi-layer design. The key intuition is that both the human annotator (user) and the machine learning algorithm are considered fallible "interpretation channels" of the real world.
SKEL comprises three main components:
- Predictor (PRED): An ensemble of learning algorithms that take sensor data as input and produce scores for potential labels, which are then aggregated into a single prediction.
- Main Algorithm: This algorithm operates in three modalities:
- Train mode: Standard supervised learning.
- Refine mode: Actively checks the quality of user answers and challenges them under specific conditions.
- Regime mode: Becomes more autonomous, only querying the user for particularly ambiguous instances.
- Knowledge Component: Stores accumulated knowledge, including:
- Stream Data (SD): Raw sensor data and labels.
- Ground Knowledge (GK): Factual knowledge about the world, implemented as a knowledge graph.
- Schematic Knowledge (SK): General knowledge, often in the form of concept hierarchies.
The novelty lies in the interactive process where the machine uses its knowledge (GK, SK) to question and refine user labels, creating a uniform confidence measure for both machine and user inputs.
Datasets
Not explicitly detailed in the paper, but the authors mention validating SKEL on "data collected in one of our experiments with students." This suggests a custom dataset collected from student participants using mobile devices.
Key Insights and Ideas
- User-generated labels are inherently noisy: The paper strongly emphasizes that non-expert users are unreliable annotators, a common issue in pervasive and lifelong learning scenarios.
- Skepticism as a virtue: The core idea of SKEL is to make the machine "skeptical" of its own predictions and the user's labels, actively seeking to verify and refine them.
- Interactive refinement is key: Simply accepting user labels is insufficient. An interactive process that allows the machine to challenge and correct labels is crucial for improving accuracy.
- Integrating model-driven and data-driven approaches: SKEL attempts to bridge the gap between data-driven machine learning and model-driven knowledge representation (GK, SK) to achieve a deeper understanding of user input.
Key Results
The paper reports a 30.7% relative improvement in performance (from F1-score of 0.26 to 0.34) on the collected student data. They also identified four common user patterns: inattentive, predictable, reliable, and tricksy users, suggesting that understanding user behavior can further enhance the system.
Limitations and Future Work
The authors acknowledge that their research is a "first attempt" at integrating model-driven and data-driven inputs. While the experimental results are promising, the generalizability of the approach to "many dimensions" is suggested as a future direction. The paper also hints at the potential for analyzing user patterns to further optimize the interactive learning process.
Summary updated