Abstract
The Dense Trajectories concept is one of the most successful approaches in action recognition, suitable for scenarios involving a significant amount of motion. However, due to noise and background motion, many generated trajectories are irrelevant to the actual human activity and can potentially lead to performance degradation. In this paper, we propose Localized Trajectories as an improved version of Dense Trajectories where motion trajectories are clustered around human body joints provided by RGB-D cameras and then encoded by local Bag-of-Words. As a result, the Localized Trajectories concept provides a more discriminative representation of actions as compared to Dense Trajectories. Moreover, we generalize Localized Trajectories to 3D by using the modalities offered by RGB-D cameras. One of the main advantages of using RGB-D data to generate trajectories is that they include radial displacements that are perpendicular to the image plane. Extensive experiments and analysis are carried out on five different datasets.
Introduction
Action recognition has long relied on the "Dense Trajectories" (DT) framework as a gold standard. By tracking points across video frames and describing the motion around them, DT captures essential temporal information. However, this approach faces two significant challenges:
- Irrelevant Noise: Uniform sampling across the entire image often captures "garbage" trajectories from background movement or camera shake, which are unrelated to the human action.
- The 2D Blind Spot: Standard DT relies on 2D optical flow. Radial motion—movement directly toward or away from the camera—results in minimal 2D displacement, making such actions difficult to describe accurately.
This paper introduces Localized Trajectories, an evolution of the DT framework that addresses these issues by making trajectories "locality-aware" and "depth-aware."
Proposed Methodology
The proposed approach iterates on the standard framework in two stages: 2D and 3D localization.
2D Localized Trajectories
Instead of treating all trajectories equally, the system utilizes 3D skeleton data from RGB-D cameras (e.g., Kinect) to group trajectories based on their proximity to human body joints.
- Joint-Based Clustering: Trajectories are assigned to the nearest human body joint, establishing an "action-motion-joint" relationship (e.g., distinguishing motion at the right hand from motion at the left foot).
- Local Bag-of-Words (BoW): Rather than a single global codebook, separate codebooks are constructed for each joint cluster. This localized representation is significantly more discriminative.
- Noise Filtering: Trajectories that fall outside a certain distance from any joint are discarded as background noise, focusing the model on relevant human motion.
3D Localized Trajectories
To overcome the limitations of 2D optical flow, the framework is extended into 3D space:
- Scene Flow: 2D optical flow is replaced with 3D "scene flow," estimating the full displacement vector field in 3D space.
- 3D Descriptors: The authors introduce 4D extensions of standard descriptors, such as the Histogram of Scene Flow (HSF), which captures both the orientation and magnitude of 3D motion, including radial displacement.
- Robust Feature Selection: Given that scene flow is inherently noisier than optical flow, a selection scheme based on classifier confidence and ambiguity is implemented. Only features that contribute to clear, reliable predictions are used for codebook generation.
Experimental Evaluation
The approach was evaluated on five diverse RGB-D datasets: MSRDailyActivity3D, Online RGB-D Action (ORGBD), G3D Gaming Action, Watch-n-Patch, and KARD.
Key Results
Localized Trajectories consistently outperformed standard Dense Trajectories across all benchmarks:
- MSRDailyActivity3D: Accuracy improved from 64.4% (Standard DT) to 74.4% (2D Localized) and 76.3% (3D Localized).
- G3D Dataset: Achieved 87.8% accuracy, compared to 80.1% for standard DT.
- Watch-n-Patch: In kitchen environments, 2D Localized Trajectories outperformed standard DT by 25.3%.
- Radial Motion Gains: For actions like "playing guitar," accuracy jumped from 20% to 70% when moving from 2D to 3D, validating the importance of depth-aware tracking.
Discussion and Conclusion
The core insight of this work is that motion context—specifically where motion occurs on the body—is vital. By filtering noise and capturing 3D radial movements, Localized Trajectories provide a more robust representation for action recognition.
While the method shows significant gains, it still faces challenges with low-motion actions and the inherent noise in current scene flow estimation. Future work will focus on automating parameter selection and generating 3D trajectories directly from raw point cloud data to further improve robustness.
Summary updated