Sensor-Based Understanding of Daily Life via Large-Scale Use of Common Sense

William Pentney, Ana-Maria Popescu, Shiaokai Wang, Henry A. Kautz, Matthai PhiliposeAAAI 2006

Introduction

Recognizing complex, goal-oriented human activities—such as "making tea" or "cleaning the bathroom"—remains a significant challenge in ubiquitous computing. While low-level physical actions like walking are easily detected via accelerometers, high-level activities require contextual understanding. Traditional supervised learning approaches are often bottlenecked by the need for massive, manually labeled datasets. This paper introduces SRCS (State Recognition using Common Sense), a system that bridges the gap between raw sensor data and high-level activity recognition by leveraging large-scale common-sense databases.

System Architecture

The SRCS system employs a multi-stage pipeline to transform discrete sensor events into probabilistic logical inferences. The process begins with a relational database of common-sense facts, which are then weighted, propositionalized into Horn clauses, and finally mapped onto a graphical model for temporal reasoning.

The SRCS multi-stage pipeline for activity recognition.

Methodology

Dense Sensing and Knowledge Integration

The system utilizes "dense sensing" via the iBracelet, a wearable RFID reader. When a user interacts with tagged household objects, the system logs the interaction. To interpret these interactions, SRCS integrates approximately 50,000 facts from the OMICS (Open Mind Indoor Common Sense) database. These facts provide the "common sense" necessary to link objects to activities (e.g., holding a toothbrush implies the activity "brushing teeth").

The iBracelet wearable RFID reader and examples of tagged household objects.

Web-Scale Validation and Probabilistic Reasoning

Because crowd-sourced databases like OMICS can be noisy, the researchers used KnowItAll to mine the web and assign weights to facts based on their frequency in online text. This "web-scale validation" filters out niche or incorrect information.

The logical structure of these facts is converted into a Markov Random Field (MRF). To account for the temporal nature of daily life, these MRFs are linked into a Chain Graph, allowing the system to maintain a persistent belief state that evolves over time. To ensure computational tractability, the system employs Query-Directed Pruning, focusing only on the local subgraph relevant to the current activities.

Experimental Results

The system was evaluated using 75 minutes of data from three users performing 15 distinct activities in a simulated home environment. Key findings include:

  • High Accuracy: The system achieved an overall accuracy of 88.42% in tracking 24 different state variables.
  • Impact of Web-Mining: Incorporating web-mined weights significantly improved precision, increasing it from 18.07% to 31.73% compared to unweighted models.
  • Inference Efficiency: Pruning the graph to a depth of $d=2$ allowed for real-time inference while maintaining a recall of 53.07%.

Discussion and Future Work

While SRCS demonstrates the power of common-sense reasoning, several limitations remain. The current model is propositional and struggles to differentiate between multiple instances of the same object type. Furthermore, the "ground truth" for activities like "making tea" can be subjective. Future work will explore integrating additional sensor modalities, such as GPS and accelerometers, and automating the selection of relevant state variables to track.

Summary updated

BibTeX
@inproceedings{pentney2006sensor,
  author = {William Pentney and Ana-Maria Popescu and Shiaokai Wang and Henry A. Kautz and Matthai Philipose},
  title = {{Sensor-Based Understanding of Daily Life via Large-Scale Use of Common Sense}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2006},
}
Figure