DeepSchema: Automatic Schema Acquisition from Wearable Sensor Data in Restaurant Situations

Eun-Sol Kim, Kyoung-Woon On, Byoung-Tak ZhangIJCAI 2016 · OpenReview

Abstract

We explore the possibility of automatically constructing hierarchical schemas from low-level sensory data. Here we suggest a hierarchical event network to build the hierarchical schemas and describe a novel machine learning method to learn the network from the data. The traditional methods for describing schemas define the primitives and the relationships between them in advance. Therefore it is difficult to adapt the constructed schemas in new situations. However, the proposed method constructs the schemas automatically from the data. Therefore, it has a novelty that the constructed schemas can be applied to new and unexpected situations flexibly. The key idea of constructing the hierarchical schema is selecting informative sensory data, integrating them sequentially and extracting high-level information. For the experiments, we collected sensory data using multiple wearable devices in restaurant situations. The experimental results demonstrate the real hierarchical schemas, which are probabilistic scripts and action primitives, constructed from the methods. Also, we show the constructed schemas can be used to predict the corresponding event to the low-level sensor data. Moreover, we show the prediction accuracy outperforms the conventional method significantly.

Introduction

In the 1970s, AI pioneers like Roger Schank introduced the concept of "scripts"—structured representations of stereotyped sequences of events, such as the steps involved in visiting a restaurant. While groundbreaking, these classical scripts were hand-coded and brittle, struggling with the messy variability of real-world environments. Modern wearable sensors offer a potential solution by providing a continuous stream of multi-modal data, yet translating this "noise" into a coherent understanding of human behavior remains a significant challenge. This paper introduces DeepSchema, a framework for automatic schema acquisition that learns the "rules" of social behavior directly from data.

Methodology: The Hierarchical Event Network

The core of DeepSchema is a three-layer Hierarchical Event Network designed to bridge the gap between raw sensory input and high-level behavioral scripts.

The Hierarchical Event Network architecture, featuring RBMs and the selective switch mechanism.

The architecture consists of:

  1. The Lower Layer (Action Units): Raw data from five sensor streams are processed by Restricted Boltzmann Machines (RBMs) to extract abstract features.
  2. The Switch Mechanism: This novel component allows the model to selectively attend to the most informative sensor streams for a given context. For instance, visual data might be prioritized during menu selection, while hand movement becomes dominant during the eating phase.
  3. The Upper Layer (Probabilistic Scripts): Action units are integrated over time to form an "Event Schema." The model then constructs a probabilistic script—a state-transition map that predicts the logical flow from one event (e.g., "Ordering") to the next (e.g., "Eating").

Experimental Setup and Data Collection

To evaluate the framework, researchers collected multi-modal data from participants dining in a real-world restaurant setting.

Wearable devices and the resulting multi-modal sensory data streams.

The data collection utilized two primary wearable devices:

  • Glass-type eye tracker: Captured first-person video and audio.
  • Watch-type wearable device: Measured physiological signals including electrodermal activity (EDA), blood volume pulse (BVP), and 3-axis acceleration.

The resulting dataset comprised seven sessions, each approximately 4,000 seconds long, annotated across 11 behavioral categories such as Greeting, Selecting a menu, Ordering, Having a meal, and Payment.

Key Results

DeepSchema demonstrated a significant improvement over traditional methods. The full model, incorporating RBMs, switch units, and temporal integration, achieved a prediction accuracy of 85.4%. This outperformed standard Logistic Regression (80.1%) and versions of the model lacking the selective switch mechanism. Notably, performance improved as the temporal window increased from 2 to 10 seconds, confirming that human behavior is best understood as a logical sequence rather than isolated snapshots.

Conclusion

DeepSchema successfully "rediscovered" the logical flow of restaurant interactions without human-defined primitives. By mimicking human sensory cue integration through its switch mechanism and incorporating physiological data, the model provides a robust path toward machines that can understand and predict complex human social scripts. Future work will focus on reducing the annotation burden and testing the architecture in less structured environments like hospitals or offices.

Summary updated

Abstract

We explore the possibility of automatically constructing hierarchical schemas from low-level sensory data. Here we suggest a hierarchical event network to build the hierarchical schemas and describe a novel machine learning method to learn the network from the data. The traditional methods for describing schemas define the primitives and the relationships between them in advance. Therefore it is difficult to adapt the constructed schemas in new situations. However, the proposed method constructs the schemas automatically from the data. Therefore, it has a novelty that the constructed schemas can be applied to new and unexpected situations flexibly. The key idea of constructing the hierarchical schema is selecting informative sensory data, integrating them sequentially and extracting high-level information. For the experiments, we collected sensory data using multiple wearable devices in restaurant situations. The experimental results demonstrate the real hierarchical schemas, which are probabilistic scripts and action primitives, constructed from the methods. Also, we show the constructed schemas can be used to predict the corresponding event to the low-level sensor data. Moreover, we show the prediction accuracy outperforms the conventional method significantly.
BibTeX
@inproceedings{kim2016deepschema,
  author = {Eun-Sol Kim and Kyoung-Woon On and Byoung-Tak Zhang},
  title = {{DeepSchema: Automatic Schema Acquisition from Wearable Sensor Data in Restaurant Situations}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
  url = {https://openreview.net/forum?id=ByEAuBf_bS},
}
Figure