Online recognition of unsegmented actions with hierarchical SOM architecture

Zahra Gharaee

Abstract

Automatic recognition of an online series of unsegmented actions requires a method for segmentation that determines when an action starts and when it ends. This paper proposes a novel approach for recognizing unsegmented actions in online test experiments using a three-layer cognitive architecture based on self-organizing neural networks. The unique features of an action sequence are represented as a series of elicited key activations by the first-layer self-organizing map (SOM). An average length of a key activation vector is calculated for all action sequences in a training set and adjusted in learning trials to generate input patterns to the second-layer SOM. These pattern vectors are clustered in the second layer, and the clusters are then labeled by an action identity in the third-layer neural network. Experimental results demonstrate that while performance drops slightly in online experiments compared to offline tests, the architecture's ability to handle unsegmented sequences makes it a practical solution for real-world scenarios.

Introduction

In computer vision, recognizing actions like a "high arm wave" or a "forward punch" is relatively straightforward when algorithms are provided with perfectly trimmed video clips. However, in real-world applications such as home robotics or smart surveillance, actions occur in a continuous, unsegmented stream. This presents the "segmentation problem": a system must determine for itself when one action ends and the next begins without the aid of manual timestamps.

This research addresses the challenge of online action recognition by proposing a biologically inspired hierarchical architecture. By mimicking biological processes like attention and event perception, the system aims to achieve high accuracy in real-time scenarios without the need for pre-trimmed data.

The Hierarchical SOM Architecture

The proposed system utilizes a three-layer cognitive architecture designed to process 3D skeleton data through a series of specialized neural layers.

1. Preprocessing and Attention

To reduce computational load and improve robustness, the system employs an attention mechanism. Instead of processing all available data, it calculates the velocity of all joints and focuses exclusively on the four most active ones. Furthermore, the data is transformed into "ego-centered" coordinates (centered on the subject's torso), making the system invariant to the subject's position or distance from the sensor.

2. Layer 1: Feature Extraction

The first layer consists of a 30x30 Self-Organizing Map (SOM) that maps high-dimensional skeleton data into a 2D representation. As an action unfolds, it leaves a "trace" of activations across this map, representing the spatial posture of the subject at any given moment.

3. The Segmentation Layer

This layer serves as the bridge between spatial postures and temporal sequences. Using a sliding window (fixed at 30 iterations), the system groups activations from Layer 1. It calculates an average length of "key activations" to package these movements into coherent patterns for the subsequent layer.

4. Layer 2: Pattern Clustering

A 35x35 SOM receives the sequences from the segmentation layer. This layer is responsible for clustering the temporal patterns into distinct action categories. Because SOMs preserve the topology of the data, similar movement trajectories are mapped to nearby neurons, allowing the system to distinguish between actions with similar postures but different temporal flows.

5. Layer 3: Labeling

The final layer is a supervised neural network that maps the clusters identified in Layer 2 to specific action labels (e.g., "Hand Catch" or "Tennis Swing").

Experimental Evaluation

The architecture was evaluated using two distinct datasets to compare performance in both controlled and real-time environments:

  • MSR-Action3D: A standard public benchmark. The study focused on a subset of 10 actions, including high throw, draw x, and tennis swing.
  • Custom Online Dataset: To validate real-time performance, the authors collected a dataset using a Kinect sensor, featuring 60 sequences of 5 different actions performed in a continuous, unsegmented flow.

Results

The researchers compared "offline" performance (using pre-segmented data) against "online" performance (using continuous streams).

Dataset Offline Accuracy (Segmented) Online Accuracy (Unsegmented)
MSR-Action3D 83% 75%
Custom Kinect Dataset 94.29% 88.74%

While there is a predictable drop in accuracy when moving from segmented to unsegmented data, the system maintained high performance. In the custom real-time test, the system achieved nearly 89% accuracy, demonstrating its viability for "in the wild" applications.

Discussion and Key Insights

A critical advantage of the hierarchical SOM approach is its ability to use temporal order as a discriminator. Actions like "lift up" and "put down" may share identical postures but differ in their sequence. The SOM architecture captures this flow without requiring complex memory units like LSTMs.

Furthermore, the system mimics human psychology by waiting for a sufficient number of "key frames" before making a classification. This "certainty threshold" ensures that the system does not prematurely misidentify an action before its unique characteristics have emerged.

Conclusion and Future Work

The proposed hierarchical SOM architecture provides a robust framework for online action recognition. While the current implementation uses a fixed window size for segmentation, which may not account for varying action speeds (e.g., a fast punch versus a slow wave), it establishes a strong foundation for real-time robotic perception. Future research will focus on implementing variable-sized sliding windows and developing prediction-error methods to automatically trigger segmentation boundaries when a movement deviates from the current predicted trajectory.

Summary updated

Abstract

Automatic recognition of an online series of unsegmented actions requires a method for segmentation that determines when an action starts and when it ends. In this paper, a novel approach for recognizing unsegmented actions in online test experiments is proposed. The method uses self-organizing neural networks to build a three-layer cognitive architecture. The unique features of an action sequence are represented as a series of elicited key activations by the first-layer self-organizing map. An average length of a key activation vector is calculated for all action sequences in a training set and adjusted in learning trials to generate input patterns to the second-layer self-organizing map. The pattern vectors are clustered in the second layer, and the clusters are then labeled by an action identity in the third layer neural network. The experiment results show that although the performance drops slightly in online experiments compared to the offline tests, the ability of the proposed architecture to deal with the unsegmented action sequences as well as the online performance makes the system more plausible and practical in real-case scenarios.
BibTeX
@inproceedings{gharaee2021online,
  author = {Zahra Gharaee},
  title = {{Online recognition of unsegmented actions with hierarchical SOM architecture}},
  booktitle = {Cogn. Process},
  year = {2021},
  url = {https://openreview.net/forum?id=xZEaEqTB4ga},
}
Figure