DeepHuMS: Deep Human Motion Signature for 3D Skeletal Sequences

Neeraj Battan, Abbhinav Venkat, Avinash Sharma

Abstract

DeepHuMS introduces a robust, deep learning-based descriptor for 3D skeletal motion retrieval. By leveraging spatio-temporal trajectory cues and a Siamese network architecture, it enables fast, accurate searching across massive datasets even when faced with noisy sensor data or varying motion speeds.

Introduction

As 3D skeletal data becomes more prevalent through motion capture (MoCap) and depth sensors like the Kinect, the ability to efficiently index and retrieve specific motions is crucial for animation, sports science, and surveillance. However, 3D motion retrieval is notoriously difficult because humans are highly articulated; the same action (like "waving") can look very different depending on the person's speed or the camera angle. Traditional methods often rely on hand-crafted features and complex optimizations that struggle with "noisy" data, such as missing joints or jitter. DeepHuMS addresses this by focusing on retrieval—finding the most similar actions in a database—rather than just recognition (labeling an action).

Overview of the DeepHuMS motion retrieval pipeline, showing the transition from query motion to embedding space and final retrieval.

Methods

The authors propose DeepHuMS (Deep Human Motion Signature), a learned embedding for 3D sequences.

Architecture and Learning

The core of the system is a Siamese network utilizing Independently Recurrent Neural Networks (IndRNN). The model processes two distinct types of input for every frame: raw 3D joint coordinates (Skeletal Features) and the displacement of joints between consecutive frames (Motion Field Features).

To train the network, a Contrastive Loss function is used to project similar motions close together in a 512-dimensional embedding space while pushing dissimilar motions far apart. A key innovation is the Similarity Metric used during training: two sequences are considered similar if they share a similar direction of movement (Motion Field) and total path traveled by joints (Motion Distance). This allows the model to learn in a self-supervised manner, discovering motion patterns without needing manual labels.

Sub-motion Retrieval

The paper also addresses the "needle in a haystack" problem: finding a short action (like a "hand raise") within a long sequence (like a "salute"). They use a secondary RNN to map short sub-sequences into the same embedding space as the full sequences, allowing for cross-length matching.

Key Results

The model was evaluated on two major benchmarks: the massive NTU RGB+D dataset (~56,000 sequences) and the high-quality HDM05 MoCap dataset.

  • Superior Accuracy: On the NTU RGB+D dataset, DeepHuMS achieved a Top-1 retrieval accuracy of 0.78, significantly outperforming other deep learning models like HCN (0.61) and standard IndRNN (0.69).
  • Efficiency: Retrieval is incredibly fast. Once the embeddings are calculated, a simple nearest-neighbor search takes only 18ms for the large NTU dataset.
  • Resilience and Invariance: By training with 20% of joints randomly missing and using speed-based data augmentation, the model proved highly effective even when sensors fail or performers move at different speeds.
Precision-Recall curves demonstrating that DeepHuMS significantly outperforms baseline methods like IndRNN and HCN on retrieval tasks.

Conclusion

DeepHuMS represents a major step forward by creating tight, distinct clusters of similar motions in an embedding space, which is essential for effective retrieval. While the authors acknowledge limitations regarding repetitive actions and precise temporal indexing, the self-supervised approach demonstrates that trajectory cues can often trump manual labels for capturing physical motion similarity.

Summary updated

BibTeX
@inproceedings{battan2019deephums,
  author = {Neeraj Battan and Abbhinav Venkat and Avinash Sharma},
  title = {{DeepHuMS: Deep Human Motion Signature for 3D Skeletal Sequences}},
  booktitle = {ACPR},
  year = {2019},
  url = {https://openreview.net/forum?id=Nf3-r-8A4B},
}
Figure