Compositional Learning of Human Activities With a Self-Organizing Neural Architecture

Luiza Mici, German Ignacio Parisi, Stefan WermterFrontiers Robotics AI 2019 · DOI · OpenReview

Introduction

For autonomous robots to function effectively in domestic environments, they must possess the ability to interpret complex human behavior. This behavior is inherently compositional: high-level activities (e.g., "making cereal") are comprised of sequences of simpler, short-term actions (e.g., "reaching," "opening," "pouring"). Recognizing these patterns from raw sensor data, such as RGB-D sequences, remains a significant challenge in robotics and AI.

The Segmentation Bottleneck

Traditional approaches to activity recognition often follow a sequential pipeline: first, the video stream is segmented into discrete action units, which are then used as input for a high-level activity classifier. This creates a "segmentation bottleneck." If the initial temporal segmentation is inaccurate—for instance, failing to distinguish the exact boundary between a "reach" and a "grasp"—the error propagates upward, frequently leading to the failure of the high-level activity recognition.

Hierarchical decomposition of the "making cereal" activity into discrete actions from an RGB-D sequence.

To overcome this, researchers have developed a system that learns to recognize both actions and activities simultaneously, making it more robust to the overlapping and fluid nature of real-world human movement.

Proposed Methodology: Hierarchical GWR Architecture

The core of the proposed system is a hierarchy of Growing When Required (GWR) networks. Unlike static neural models, GWR networks are self-organizing and dynamic; they begin with a minimal structure and "grow" new neurons only when they encounter input data that cannot be adequately represented by the existing network.

The architecture is organized into four primary layers:

  1. Input Processing ($GWR_b$ and $GWR_o$): These layers handle low-level features, specifically 3D skeletal body poses and object identities/textures.
  2. Action Recognition ($GWR_a$): This layer integrates body and object data over a short temporal window to identify transitive actions.
  3. Activity Recognition ($GWR_A$): The top layer integrates the outputs of the action layer over a longer duration to classify the overall high-level activity.
The proposed hierarchical neural framework, featuring specialized GWR layers for body pose, object recognition, actions, and activities.

Top-Down Growth Modulation

A key innovation in this work is the top-down modulation mechanism. Standard GWR networks grow based on "quantization error" (how well a neuron matches the input). The authors introduced a feedback signal based on "classification error." If the network accurately represents the data but assigns an incorrect label, the feedback triggers the growth of a new neuron. This forces the network to become more specialized in regions of the feature space where different activities are easily confused, effectively using labels as constraints to guide neural growth.

Experimental Evaluation

The model was evaluated using the CAD-120 (Cornell Activity Dataset), a rigorous benchmark containing 120 videos of 10 long-duration daily activities performed by four different actors. The dataset is characterized by significant occlusions, varying movement styles, and fine-grained sub-activities that often overlap in time.

Key Insights and Results

The study yielded several critical findings:

  • State-of-the-Art Performance: The hierarchical model achieved 92.0% accuracy in high-level activity recognition without requiring ground-truth temporal segmentation. This significantly outperformed traditional models like Conditional Random Fields (CRFs) and Support Vector Machines (SVMs), which typically achieve 80-83% accuracy.
  • Structural Efficiency: The top-down modulation mechanism proved highly efficient. By focusing growth on discriminative features rather than raw data density, the number of neurons in the action-level network ($GWR_a$) was reduced by approximately 50% (from 1,500 to 750) without sacrificing performance.
  • Robustness to Sub-Action Ambiguity: Qualitative analysis demonstrated that the model could correctly identify the "big picture" activity even when it struggled to precisely label every individual sub-action. This suggests the hierarchical structure successfully captures the underlying intent of the human actor.

Discussion and Future Work

While the system excels at high-level recognition, its accuracy on individual sub-activities was lower than some fully supervised state-of-the-art methods. This is likely due to the visual similarity between actions like "reaching" and "moving" in the feature space.

Future research will focus on incorporating audio cues to resolve visual ambiguities and transitioning from recognition to action strategy generation, enabling robots to proactively assist humans in daily tasks.

Summary updated

Abstract

An important step for assistive systems and robot companions operating in human environments is to learn the compositionality of human activities, i.e, recognize both activities and their comprising actions.Most existing approaches address action and activity recognition as separate tasks, i.e., actions need to be inferred before the activity labels, and are thus highly sensitive to the correct temporal segmentation of the activity sequences.In this paper, we present a novel learning approach that jointly learns human activities on two levels of semantic and temporal complexity:~1)~transitive actions such as reaching and opening, e.g. a cereal box, and 2) high-level activities such as having breakfast. Our model consists of a hierarchy of GWR networks which process and learn inherent spatiotemporal dependencies of multiple visual cues extracted from the human body skeletal representation and the interaction with objects.The neural architecture learns and semantically segments input RGB-D sequences of high-level activities into their composing actions, without supervision. We investigate the performance of our architecture with a set of experiments on a publicly available benchmark dataset.The experimental results show that our approach outperforms the state of the art with respect to the classification of the high-level activities.Additionally, we introduce a novel top-down modulation mechanism to the architecture which uses the actions and activity labels as constraints during the learning phase.In our experiments, we show how this mechanism can be used to control the network's neural growth without decreasing the overall performance.
BibTeX
@article{mici2019compositional,
  author = {Luiza Mici and German Ignacio Parisi and Stefan Wermter},
  title = {{Compositional Learning of Human Activities With a Self-Organizing Neural Architecture}},
  journal = {Frontiers Robotics AI},
  year = {2019},
  doi = {10.3389/frobt.2019.00072},
}
Figure