Online recognition of unsegmented actions with hierarchical SOM architecture

Zahra Gharaee

In the world of computer vision, recognizing a "high arm wave" or a "forward punch" is relatively straightforward if you give an algorithm a perfectly trimmed video clip. But in the real world—think home robotics or smart surveillance—actions don't come with "start" and "stop" timestamps. They happen in a continuous, messy stream.

A recent paper by Zahra Gharaee, titled "Online recognition of unsegmented actions with hierarchical SOM architecture," proposes a biologically inspired way to solve this "segmentation problem" using a multi-layered neural network architecture.

TL;DR

The researchers developed a three-layer hierarchical architecture based on Self-Organizing Maps (SOMs) to recognize human actions from a continuous, unsegmented stream of 3D skeleton data. By mimicking biological processes like attention and event perception, the system achieves high accuracy in real-time scenarios without needing pre-trimmed video clips.

What Problem Does This Paper Address?

Most action recognition models are "offline." They are trained and tested on datasets where each video contains exactly one action, neatly clipped from beginning to end. However, a robot interacting with a human needs to perform "online" recognition—it must watch a continuous stream of movement and figure out for itself when one action ends and the next begins.

This is the segmentation problem. It’s difficult because different people perform actions at different speeds, and many actions (like "using a hammer" vs. "forward punch") might start with the exact same initial movements. The challenge is to create a system that is robust to these variations and can make decisions in real-time with minimal delay.

Methods

The authors propose a three-layer cognitive architecture inspired by how humans perceive events:

  1. Preprocessing & Attention: Instead of processing every bit of data, the system uses an attention mechanism. It calculates the velocity of all joints and focuses only on the four most active ones. It also transforms the data into "ego-centered" coordinates (centered on the person's stomach) to make the system invariant to where the person is standing or how far they are from the camera.
  2. Layer 1 (Feature Extraction): A 30x30 SOM maps the high-dimensional skeleton data into a 2D representation. As an action unfolds, it leaves a "trace" of activations across this map.
  3. Segmentation Layer: This is the "secret sauce." The system uses a sliding window (fixed at 30 iterations) to group these activations. It calculates an average length of "key activations" to determine how to package these movements for the next layer.
  4. Layer 2 (Pattern Clustering): A 35x35 SOM takes the sequences from Layer 1 and clusters them into distinct action categories.
  5. Layer 3 (Labeling): A simple supervised neural network layer that maps the clusters in Layer 2 to specific action names (e.g., "Hand Catch").

What’s unique here is the use of SOMs. Unlike standard deep learning "black boxes," SOMs preserve the topology of the data, meaning similar postures are mapped to nearby neurons, allowing the system to "see" the trajectory of an action across the neural lattice.

Datasets

The architecture was evaluated on two main datasets:

  • MSR-Action3D: A standard public benchmark containing 20 different actions performed by 10 subjects. For this study, the authors focused on a subset of 10 actions (like high throw, draw x, and tennis swing).
  • Custom Online Dataset: To validate real-time performance, the authors collected a new dataset using a Kinect sensor, featuring 60 sequences of 5 different actions performed in a continuous, unsegmented flow.

Key Insights and Ideas

  • Temporal Order as a Discriminator: The authors point out that some actions, like "lift up" and "put down," have identical postures but in reverse order. The hierarchical SOM naturally captures this temporal flow without needing a complex "memory" unit like an LSTM.
  • The "Aha" Moment of Certainty: The system doesn't just guess instantly. It waits until it perceives a sufficient number of "key frames." This mimics human psychology—we often can't tell if someone is reaching for a watch or starting a wave until the movement progresses past a certain point.
  • Bio-inspired Efficiency: By using an attention mechanism to ignore "quiet" joints, the system significantly reduces the computational load, making it much more plausible for low-power robotic systems.

Key Results

The researchers compared "offline" performance (on pre-segmented data) against "online" performance (on continuous streams):

Dataset Offline Accuracy (Segmented) Online Accuracy (Unsegmented)
Experiment 1 (MSR-Action3D) 83% 75%
Experiment 2 (Custom Kinect) 94.29% 88.74%

While there is a slight drop in accuracy when moving to unsegmented data, the results are impressive. The system was able to correctly categorize actions within just a few iterations of the movement starting. In Experiment 2, the system achieved nearly 89% accuracy in a true real-time, "in the wild" test.

Limitations and Future Work

The primary limitation identified is the fixed window size ($T$) for segmentation. Currently, the system uses a fixed length of 30 iterations. However, a "forward punch" is much faster than "drawing a circle."

The authors suggest that future versions should implement a variable-sized sliding window that adapts to the speed of the action. They also propose developing a method to calculate "prediction error"—if the next movement doesn't match the predicted trajectory of the current action, the system would automatically trigger a "segmentation boundary" to start recognizing a new action.

Summary updated

Abstract

Automatic recognition of an online series of unsegmented actions requires a method for segmentation that determines when an action starts and when it ends. In this paper, a novel approach for recognizing unsegmented actions in online test experiments is proposed. The method uses self-organizing neural networks to build a three-layer cognitive architecture. The unique features of an action sequence are represented as a series of elicited key activations by the first-layer self-organizing map. An average length of a key activation vector is calculated for all action sequences in a training set and adjusted in learning trials to generate input patterns to the second-layer self-organizing map. The pattern vectors are clustered in the second layer, and the clusters are then labeled by an action identity in the third layer neural network. The experiment results show that although the performance drops slightly in online experiments compared to the offline tests, the ability of the proposed architecture to deal with the unsegmented action sequences as well as the online performance makes the system more plausible and practical in real-case scenarios.
BibTeX
@inproceedings{gharaee2021online,
  author = {Zahra Gharaee},
  title = {{Online recognition of unsegmented actions with hierarchical SOM architecture}},
  booktitle = {Cogn. Process},
  year = {2021},
  url = {https://openreview.net/forum?id=xZEaEqTB4ga},
}

Loading related content...

Figure