Towards Cooperative Self-adapting Activity Recognition

Andreas Jahn, Sven Tomforde, Michel Morold, Klaus David, Bernhard Sick

TL;DR

This paper proposes a novel approach to activity recognition (AR) by enabling AR systems to cooperate and learn from each other. This collaboration aims to create "smart" AR systems that can adapt to individual users and their environments in real-time, improving accuracy and reducing energy consumption.

What Problem Does This Paper Address?

Activity Recognition (AR) systems aim to understand human activities and contexts using sensor data from devices like smartphones. While AR is a well-established field, current systems face several limitations:

  • Fixed Sensor Configurations: They rely on a predefined set of sensors available on a device.
  • Lack of User-Specific Adaptation: Pre-trained models are often not tailored to individual user behaviors, leading to suboptimal performance.
  • Static Activity Repertoires: The set of activities an AR system can recognize is fixed at design time and cannot be easily extended.
  • Underutilization of Information: AR systems typically operate in isolation, failing to leverage the vast amount of information available from other nearby AR systems or even human input.

The authors argue that by enabling cooperation between AR systems and incorporating human feedback, a new generation of "smart" AR systems can emerge that are more flexible, accurate, and energy-efficient.

Methods

The core of the proposed approach is a cooperative and self-adapting AR system built upon the principles of Organic Computing (OC). The architectural blueprint is based on the Observer/Controller (O/C) tandem, adapted for cooperating AR systems.

The system is structured into two main layers:

  • Reaction Layer: This layer handles immediate reactions to observed behavior and aims to improve recognition accuracy and reduce energy consumption through cooperation. It consists of:
  • Observer Component: This component features a four-stage recognition chain:
  • Pre-processing and Feature Extraction: Raw sensor data is cleaned, segmented, and features are extracted.
  • Recognition of Motion Primitives: Basic movements (e.g., arm lifting) are identified using algorithms like SVM or kNN.
  • Recognition of Low-Level Activities: Temporally coherent sequences of motion primitives (e.g., writing) are recognized, often using Hidden Markov Models (HMMs).
  • Recognition of High-Level Activities: Related low-level activities and external information are combined to identify complex activities (e.g., participating in a meeting).
  • Controller Component: Triggers actuators based on recognized activities.
  • Cooperative Reaction Component: Collects and integrates information from other AR systems and sources to enhance recognition.
  • Adaptation Layer: This layer focuses on long-term improvements by adapting the Reaction Layer. It monitors the Reaction Layer's performance and modifies its behavior to achieve user-specific adaptation and the detection of novel activities. It also includes a cooperative adaptation module that gathers information from other AR systems and humans.

The paper emphasizes the importance of cooperation at multiple levels of abstraction (motion primitives, low-level activities, high-level activities) and the potential for active learning to efficiently acquire user knowledge.

Datasets

Not discussed in the paper.

Key Insights and Ideas

  • Cooperation as a Paradigm Shift: The paper argues that moving from isolated AR systems to cooperative ones is crucial for overcoming current limitations and enabling truly "smart" systems.
  • Multi-Level Cooperation: Cooperation is not limited to a single stage but can occur at the level of motion primitives, low-level activities, and high-level activities, offering a comprehensive approach.
  • Leveraging External Information: The system actively seeks to integrate information from other AR systems, sensors, and human users, treating them as valuable knowledge sources.
  • Self-Adaptation and Self-Extension: The goal is to create systems that can adapt to individual users' behaviors and even discover and learn new activities autonomously.
  • Organic Computing Principles: The adoption of OC principles provides a robust framework for designing systems that can self-organize, self-assess, and self-adapt at runtime.

Key Results

The paper presents a conceptual framework and a research roadmap rather than empirical results. The key "results" are the proposed architectural blueprint and the identified research challenges that need to be addressed to realize the vision of cooperative, self-adapting AR systems. These challenges include:

  • Assessing and selecting information available through cooperation.
  • Improving classification accuracy via cooperation.
  • Achieving energy efficiency through cooperation.
  • Enabling short-term adaptation of the Reaction Layer.
  • Runtime customization of AR systems.
  • Runtime self-extension for detecting novel activities.
  • Cooperation with human users for long-term improvement.
  • Cooperation with other AR systems for long-term improvement.
  • Estimating the success of cooperation.

Limitations and Future Work

The paper acknowledges that the proposed architecture is a blueprint and that significant research challenges remain. The authors explicitly outline these challenges in Section 5, which represent the future work directions. These include developing robust methods for information assessment, cooperative learning, energy management, and seamless integration of human feedback. The paper also highlights the need for scenarios and evaluation metrics to empirically prove the benefits of cooperation.

Summary updated

BibTeX
@inproceedings{jahn2018towards,
  author = {Andreas Jahn and Sven Tomforde and Michel Morold and Klaus David and Bernhard Sick},
  title = {{Towards Cooperative Self-adapting Activity Recognition}},
  booktitle = {PECCS},
  year = {2018},
  doi = {10.5220/0006856102150222},
}
Figure