Unsupervised Analysis of Human Gestures

Tianshu Wang, Heung-Yeung Shum, Ying-Qing Xu, Nanning ZhengIEEE Pacific Rim Conference on Multimedia 2001 · DOI

TL;DR

This paper introduces an unsupervised method to automatically segment and cluster human gesture sequences into meaningful "primitive gestures" without prior knowledge. By treating gestures like natural language, they extract a "lexicon" of basic movements, enabling more efficient and scalable gesture analysis for applications like human-computer interaction and video indexing.

What Problem Does This Paper Address?

Recognizing human gestures from video is crucial for various applications, but current methods typically require large, manually labeled datasets of gestures. This manual process is time-consuming, error-prone, and impractical for scenarios where the set of gestures isn't known beforehand. The paper aims to overcome this by developing an unsupervised approach that can automatically discover and represent basic gesture components from continuous motion sequences, much like how words form sentences in natural language.

Methods

The proposed approach consists of three main steps:

  1. Temporal Segmentation: The continuous stream of human motion is first broken down into very fine-grained "atomic" segments. This is an over-segmentation process, meaning each segment is a basic movement that might not have inherent meaning on its own. The segmentation identifies points of natural inconsistency in movement, such as changes in velocity or direction.
  2. Clustering by Hidden Markov Models (HMMs): These atomic segments are then clustered into a predefined number of groups using HMMs. HMMs are well-suited for modeling dynamic processes and provide a distance metric for comparing gesture sequences. Hierarchical clustering is employed to group similar segments, with each cluster representing a distinct type of atomic gesture. This step transforms the continuous observation into a discrete symbol sequence, where each segment is replaced by its cluster label.
  3. Extracting Lexicon (COMPRESSIVE Algorithm): The discrete symbol sequence is then analyzed to infer a "lexicon" of primitive gestures. This is analogous to learning words from text. The paper uses a heuristic algorithm called COMPRESSIVE, which identifies recurring patterns (potential "words") that provide the highest compression ratio for the sequence. This process effectively learns a compact representation of the gesture data.

Datasets

The authors conducted their experiments on a dataset of musical conducting gestures recorded from a professional conductor. The dataset comprises approximately 8 minutes of motion data, resulting in 163 segments after temporal segmentation. The paper does not introduce a new benchmark dataset but uses this specific recording for evaluation.

Key Insights and Ideas

  • Gestures as Language: The core idea of treating human gestures as a form of natural language, with primitive gestures acting as "words" and their temporal ordering forming "grammar," is a powerful analogy that drives the unsupervised learning approach.
  • Unsupervised Discovery of Primitives: The ability to automatically discover and cluster basic gesture components without any prior labels is a significant step towards more scalable and adaptable gesture recognition systems.
  • COMPRESSIVE for Lexicon Extraction: The use of a compression-based algorithm (COMPRESSIVE) to infer a lexicon from discrete symbol sequences is an innovative application of data compression techniques to gesture analysis. It highlights how finding repetitive patterns can reveal underlying structure.
  • Hierarchical Structure: The learned lexicon can have a hierarchical structure, where more complex gestures are composed of simpler, learned primitive gestures, mirroring linguistic structures.

Key Results

  • The approach successfully segmented and labeled continuous human gestures.
  • In their experiment on musical conducting, the algorithm extracted 8 "words" (primitive gestures), including two-beat, three-beat, four-beat, and six-beat patterns.
  • The five-beat pattern was not extracted as a distinct primitive due to fewer occurrences, but was instead subdivided.
  • Out of 163 segments, 104 were correctly labeled as known patterns, while the remaining were assigned as new patterns.
  • The entire learning process for the 8-minute sequence took approximately 4 minutes on a PIII 500MHz CPU.

Limitations and Future Work

  • Frequency Dependence: The current approach relies on the frequency of gestures. It is most suitable for structured human motion where gestures are repeated many times.
  • Potential Words vs. Definite Lexicon: The algorithm extracts "potential words" rather than a definite lexicon, meaning it identifies recurring patterns that are likely meaningful but doesn't guarantee they correspond to human-defined gesture categories.
  • Future Work: The authors suggest extending the approach to general human gestures (beyond structured motion) and applying it to partially labeled data. They also mention the possibility of adapting the approach to incremental and online frameworks.

Summary updated

BibTeX
@inproceedings{wang2001unsupervised,
  author = {Tianshu Wang and Heung-Yeung Shum and Ying-Qing Xu and Nanning Zheng},
  title = {{Unsupervised Analysis of Human Gestures}},
  booktitle = {IEEE Pacific Rim Conference on Multimedia},
  year = {2001},
  doi = {10.1007/3-540-45453-5\_23},
}
Figure