TL;DR
This paper introduces Multiresolution Match Kernels (MMK), a novel framework for gesture video classification that improves upon the traditional Bag-of-Features (BoF) method by incorporating spatio-temporal information at multiple resolutions. MMK offers a more discriminative feature representation, leading to significantly higher recognition accuracies, especially for gestures with similar movements but different temporal ordering.
What Problem Does This Paper Address?
Gesture recognition from video data has seen a resurgence with the advent of depth sensors like the Microsoft Kinect, enabling the use of RGB-D data. A common approach is the Bag-of-Features (BoF) model, which extracts local features (e.g., SIFT, HOG) and represents videos as histograms of these features. However, BoF has two main drawbacks: it provides a coarse similarity measure due to feature quantization and loses crucial spatio-temporal information by pooling features globally. This makes it difficult to distinguish between gestures that have similar local movements but differ in their execution order or precise location in space and time.
Methods
The authors propose the Multiresolution Match Kernel (MMK), which builds upon the efficient match kernel concept and spatio-temporal pyramids. Instead of simply counting feature occurrences (as in BoF), MMK uses kernel functions to measure descriptor similarity, providing a finer-grained measure. The core innovation is the integration of a spatio-temporal pyramid structure. Videos are recursively divided into smaller cuboids (voxels) at multiple levels of resolution. MMK then computes kernel-based similarities between corresponding voxels across different videos and resolutions.
To address the computational cost of this multiresolution approach, the authors derive a cost-effective formulation by working in a mapped feature space. This involves estimating feature maps induced by the kernels at each resolution and concatenating them to form a comprehensive video representation. The MMK framework can be seen as a linear combination of kernels, where higher resolutions (finer levels of the pyramid) are given more weight.
Datasets
The proposed method is evaluated on the MSRGesture3D dataset. This dataset contains 12 depth gesture videos representing American Sign Language letters and words, performed by 10 users, with a total of 336 examples.
Key Insights and Ideas
- Beyond Global Pooling: MMK moves beyond the global pooling of BoF by explicitly considering spatio-temporal relationships at multiple scales, leading to a richer video representation.
- Kernelized Similarity: Replacing the hard assignment in BoF with kernel functions allows for a more nuanced and discriminative similarity measure between feature descriptors.
- Multiresolution Advantage: The performance consistently improves with increasing pyramid resolution (MMK-1, MMK-2, MMK-3), highlighting the benefit of capturing gestures at different levels of detail.
- Handling Temporal Invariance: MMK is shown to be more effective than BoF for gestures with similar movements but different temporal ordering, a key limitation of the standard BoF.
Key Results
- MMK significantly outperforms the standard Bag-of-Features (BoF) method across three different feature descriptors (SIFT, LBP, and HOG).
- For SIFT features, MMK-3 (using 3 levels of spatio-temporal pyramid) achieved an accuracy of 91.66% compared to 71.28% for BoF. Similar improvements were observed for LBP and HOG.
- When evaluated using a leave-one-subject-out approach, MMK-3 achieved accuracies of 94.6% (SIFT), 94.1% (LBP), and 91.5% (HOG), surpassing existing state-of-the-art results on the MSRGesture3D dataset.
- The confusion matrix analysis indicated that MMK-3 was effective, though some gestures (like "Milk") were still occasionally misclassified, suggesting areas for further improvement.
Limitations and Future Work
- Computational Cost: While the mapped feature space formulation significantly improves efficiency, estimating MMKs can still be computationally intensive, especially for very deep pyramids or large dictionaries.
- Specific Gesture Misclassification: The paper notes that certain gestures, like "Milk," which involve a clenched fist similar to other gestures, were still prone to misclassification.
- Future Work: The authors plan to extend the MMK method to other video-based classification tasks to study its generalizability. They also aim to investigate ways to overcome challenges like the misclassification of gestures with similar underlying movements.
Summary updated