Multiresolution Match Kernels for Gesture Video Classification

Hemanth Venkateswara, Vineeth N. Balasubramanian, Prasanth Lade, Sethuraman PanchanathanWorkshops @ ICME 2013 · OpenReview

TL;DR

This paper introduces Multiresolution Match Kernels (MMK), a novel framework for gesture video classification that improves upon the traditional Bag-of-Features (BoF) method by incorporating spatio-temporal information at multiple resolutions. MMK offers a more discriminative feature representation, leading to significantly higher recognition accuracies, especially for gestures with similar movements but different temporal ordering.

What Problem Does This Paper Address?

Gesture recognition from video data has seen a resurgence with the advent of depth sensors like the Microsoft Kinect, enabling the use of RGB-D data. A common approach is the Bag-of-Features (BoF) model, which extracts local features (e.g., SIFT, HOG) and represents videos as histograms of these features. However, BoF has two main drawbacks: it provides a coarse similarity measure due to feature quantization and loses crucial spatio-temporal information by pooling features globally. This makes it difficult to distinguish between gestures that have similar local movements but differ in their execution order or precise location in space and time.

Methods

The authors propose the Multiresolution Match Kernel (MMK), which builds upon the efficient match kernel concept and spatio-temporal pyramids. Instead of simply counting feature occurrences (as in BoF), MMK uses kernel functions to measure descriptor similarity, providing a finer-grained measure. The core innovation is the integration of a spatio-temporal pyramid structure. Videos are recursively divided into smaller cuboids (voxels) at multiple levels of resolution. MMK then computes kernel-based similarities between corresponding voxels across different videos and resolutions.

To address the computational cost of this multiresolution approach, the authors derive a cost-effective formulation by working in a mapped feature space. This involves estimating feature maps induced by the kernels at each resolution and concatenating them to form a comprehensive video representation. The MMK framework can be seen as a linear combination of kernels, where higher resolutions (finer levels of the pyramid) are given more weight.

Datasets

The proposed method is evaluated on the MSRGesture3D dataset. This dataset contains 12 depth gesture videos representing American Sign Language letters and words, performed by 10 users, with a total of 336 examples.

Key Insights and Ideas

  • Beyond Global Pooling: MMK moves beyond the global pooling of BoF by explicitly considering spatio-temporal relationships at multiple scales, leading to a richer video representation.
  • Kernelized Similarity: Replacing the hard assignment in BoF with kernel functions allows for a more nuanced and discriminative similarity measure between feature descriptors.
  • Multiresolution Advantage: The performance consistently improves with increasing pyramid resolution (MMK-1, MMK-2, MMK-3), highlighting the benefit of capturing gestures at different levels of detail.
  • Handling Temporal Invariance: MMK is shown to be more effective than BoF for gestures with similar movements but different temporal ordering, a key limitation of the standard BoF.

Key Results

  • MMK significantly outperforms the standard Bag-of-Features (BoF) method across three different feature descriptors (SIFT, LBP, and HOG).
  • For SIFT features, MMK-3 (using 3 levels of spatio-temporal pyramid) achieved an accuracy of 91.66% compared to 71.28% for BoF. Similar improvements were observed for LBP and HOG.
  • When evaluated using a leave-one-subject-out approach, MMK-3 achieved accuracies of 94.6% (SIFT), 94.1% (LBP), and 91.5% (HOG), surpassing existing state-of-the-art results on the MSRGesture3D dataset.
  • The confusion matrix analysis indicated that MMK-3 was effective, though some gestures (like "Milk") were still occasionally misclassified, suggesting areas for further improvement.

Limitations and Future Work

  • Computational Cost: While the mapped feature space formulation significantly improves efficiency, estimating MMKs can still be computationally intensive, especially for very deep pyramids or large dictionaries.
  • Specific Gesture Misclassification: The paper notes that certain gestures, like "Milk," which involve a clenched fist similar to other gestures, were still prone to misclassification.
  • Future Work: The authors plan to extend the MMK method to other video-based classification tasks to study its generalizability. They also aim to investigate ways to overcome challenges like the misclassification of gestures with similar underlying movements.

Summary updated

Abstract

The emergence of depth imaging technologies like the Microsoft Kinect has renewed interest in computational methods for gesture classification based on videos. For several years now, researchers have used the Bag-of-Features (BoF) as a primary method for generation of feature vectors from video data for recognition of gestures. However, the BoF method is a coarse representation of the information in a video, which often leads to poor similarity measures between videos. Besides, when features extracted from different spatio-temporal locations in the video are pooled to create histogram vectors in the BoF method, there is an intrinsic loss of their original locations in space and time. In this paper, we propose a new Multiresolution Match Kernel (MMK) for video classification, which can be considered as a generalization of the BoF method. We apply this procedure to hand gesture classification based on RGB-D videos of the American Sign Language(ASL) hand gestures and our results show promise and usefulness of this new method.
BibTeX
@inproceedings{venkateswara2013multiresolution,
  author = {Hemanth Venkateswara and Vineeth N. Balasubramanian and Prasanth Lade and Sethuraman Panchanathan},
  title = {{Multiresolution Match Kernels for Gesture Video Classification}},
  booktitle = {Workshops @ ICME},
  year = {2013},
  url = {https://openreview.net/forum?id=Eq0nzKagvE7},
}
Figure