Human Behavior Cognition Using Smartphone Sensors

Ling Pei, Robert Guinness, Ruizhi Chen, Jingbin Liu, Heidi Kuusniemi, Yuwei Chen, Liang Chen, Jyrki Kaistinen2013 · DOI

TL;DR

This paper introduces the LoMoCo (Location-Motion-Context) model, a novel approach to inferring human behavior and context using smartphone sensors. By combining ubiquitous positioning and motion recognition, the system achieves activity-level understanding with high accuracy, paving the way for more intelligent cognitive phone applications.

What Problem Does This Paper Address?

The increasing ubiquity of smartphones, packed with a diverse array of sensors, presents an exciting opportunity for developing "cognitive" platforms that can understand and infer human behavior and context. While significant research has focused on individual aspects like positioning or motion recognition, a unified framework to integrate these capabilities for comprehensive human behavior modeling has been lacking. Specifically, the challenge lies in bridging the gap from raw sensor data to rich contextual understanding, enabling applications to answer questions like "What is the user doing?" or "What is their current situation?" in natural environments, both indoors and outdoors.

Methods

The paper proposes a three-tiered architecture for sensing social context and modeling human behavior:

  1. Sensing Social Context: This module leverages navigation and audio/visual sensors to capture raw data, which is then processed into physical parameters (e.g., position, acceleration) and further refined into features/patterns. This forms the foundation of their "Context Pyramid."
  2. Social Context Analysis and Human Behavior Modeling: This module builds upon the lower levels of the Context Pyramid to infer higher-level descriptors, including simple contexts (location, motion), activity-level descriptors, and ultimately, rich context. The core of this module is the LoMoCo (Location-Motion-Context) model.
  3. Smartphone-based Social Applications: This layer utilizes the human behavior models developed in Module II to power various applications.

The LoMoCo model is central to this research. It combines location patterns (Lo) and motion patterns (Mo) to infer a context (Co) using Bayesian reasoning. The model is trained on sensor data and then used to predict the most probable context given observed features. For motion recognition, they extract thirteen features from accelerometer, gyroscope, and magnetometer data and employ a Least Squares Support Vector Machine (LS-SVM) classifier, chosen for its high accuracy and computational efficiency compared to standard SVMs. For positioning, they integrate satellite-based, sensor-based, and network-based solutions for seamless indoor/outdoor coverage, utilizing WiFi fingerprinting for indoor localization.

Datasets

The paper describes experiments conducted in the Finnish Geodetic Institute (FGI) office.

  • Positioning: A dataset was collected using an Android WiFi fingerprinting application on a Samsung Galaxy Nexus. 43 reference points were used to generate a radio map. Sixty samples per reference point were collected from four directions, with a total of 560 samples used for accuracy verification.
  • Motion Recognition: Four testers participated in data collection over five days in the FGI office. Each tester performed six defined motion states, and over 1,200 samples were collected per tester.
  • Human Behavior Modeling (LoMoCo): For training, twenty context samples were collected, with four samples for each of five predefined contexts (C1-C5). For testing, sixty-seven context samples were collected, including fourteen C1, fifteen C2, ten C4, eight C3, and fifteen C5 samples, along with five abnormal contexts.

No new benchmark datasets were explicitly introduced, but the experiments were conducted in a real-world office environment.

Key Insights and Ideas

  • The Context Pyramid: Abstracting context into a hierarchical pyramid (Raw Sensor Data to Rich Context) provides a structured way to think about and build context-aware systems.
  • LoMoCo Model: The integration of location and motion information is crucial for robust human behavior modeling. Neither alone is sufficient; their synergy unlocks higher accuracy.
  • LS-SVM for Motion Recognition: LS-SVM offers a compelling alternative to traditional SVMs for motion recognition, achieving high accuracy with reduced computational complexity.
  • Seamless Indoor/Outdoor Positioning: Combining multiple positioning technologies (GNSS, WiFi, sensors) is essential for continuous location awareness in diverse environments.
  • Battery Efficiency Considerations: The paper highlights the significant impact of sensor usage on battery life, suggesting strategies like turning off GPS indoors or reducing WiFi scanning rates.

Key Results

  • Motion Recognition Accuracy: The LS-SVM classifier achieved an accuracy rate of up to 92.9% for motion state recognition.
  • Positioning Accuracy: Location accuracy was reported as 1.9 meters in corridors, 2.7 meters in semi-open spaces, and 3.5 meters in the main lobby/break room. These accuracies are deemed sufficient for room-level activity recognition.
  • LoMoCo Model Performance: The LoMoCo model achieved an Activity-Level Descriptor level with a correct context recognition rate of 90.3% when both location and motion features were used.
  • Feature Importance: When using only location features, context recognition was 85.5% accurate, while using only motion features resulted in lower accuracy (specific percentage not explicitly stated for this case, but Table 7 shows varying results for different contexts).
  • Battery Drain: Enabling motion sensors, WiFi, and GPS simultaneously resulted in the highest battery drain rate of 27.5% h⁻¹. Using only motion sensors and WiFi allowed for approximately 8.3 hours of operation on a fully charged battery.

Limitations and Future Work

  • Motion State Definition and Feature Selection: The authors acknowledge that motion state definitions and feature selection can be case-specific. They note that confusions between sharp and gradient turning might not impact classification if merged. Future work will focus on optimizing motion definitions and feature selection for more effective motion recognition.
  • Undefined Contexts: The current LoMoCo model struggles with undefined or abnormal contexts. Future work aims to improve the model's ability to detect such behaviors.
  • Group Behavior: The current research focuses on individual behavior. Future work will expand to study social activities involving groups of people.
  • Rich Context: The ultimate goal is to reach the "Rich Context" level, which will involve modeling more complex human behavior, including psychological states and social media context.

Summary updated

BibTeX
@misc{pei2013human,
  author = {Ling Pei and Robert Guinness and Ruizhi Chen and Jingbin Liu and Heidi Kuusniemi and Yuwei Chen and Liang Chen and Jyrki Kaistinen},
  title = {{Human Behavior Cognition Using Smartphone Sensors}},
  year = {2013},
  doi = {10.3390/s130201402},
}
Figure