Localized Trajectories for 2D and 3D Action Recognition

Konstantinos Papadopoulos, Girum G. Demisse, Enjie Ghorbel, Michel Antunes, Djamila Aouada, Björn E. Ottersten

TL;DR

This paper introduces Localized Trajectories, an evolution of the popular Dense Trajectories (DT) framework for action recognition. By clustering motion trajectories around human body joints and extending the analysis into 3D using scene flow, the authors significantly reduce background noise and capture "radial" movements (motion toward or away from the camera) that traditional 2D methods often miss.

What Problem Does This Paper Address?

For years, "Dense Trajectories" have been a gold standard in action recognition. The idea is simple: track points across video frames and describe the motion around them. However, this approach has two major flaws:

  1. Irrelevant Noise: Because it samples points uniformly across the entire image, it picks up a lot of "garbage" trajectories from background movement or camera shake that have nothing to do with the human action.
  2. The 2D Blind Spot: Standard DT relies on 2D optical flow. If someone moves their hand directly toward the camera (radial motion), the 2D displacement is tiny, making the action nearly invisible to the descriptor.

The authors aim to fix these by making trajectories "locality-aware" and "depth-aware."

Methods

The paper proposes two main iterations: 2D Localized Trajectories and 3D Localized Trajectories.

1. 2D Localized Trajectories

Instead of treating all trajectories equally, the authors use 3D skeleton data (provided by RGB-D cameras like the Kinect) to group trajectories.

  • Joint-Based Clustering: Trajectories are assigned to the nearest human body joint. This creates a relationship of "action-motion-joint" (e.g., knowing that a specific motion happened at the right hand vs. the left foot).
  • Local Bag-of-Words (BoW): Instead of one giant codebook for the whole video, they build a separate codebook for each joint cluster. This makes the representation much more discriminative.
  • Noise Filtering: Trajectories too far from any joint are discarded as background noise.

2. 3D Localized Trajectories

To solve the radial motion problem, they move from 2D to 3D:

  • Scene Flow: They replace 2D optical flow with 3D "scene flow," which estimates the displacement vector field in 3D space.
  • 3D Descriptors: They introduce 4D extensions of standard descriptors, such as the Histogram of Scene Flow (HSF), which captures both the orientation and magnitude of 3D motion.
  • Robust Feature Selection: Since scene flow is naturally noisier than optical flow, they implement a selection scheme based on classifier confidence and ambiguity. They only use features that help the classifier make clear, reliable predictions to build their codebooks.

Datasets

The authors evaluated their approach on five diverse RGB-D datasets:

  • MSRDailyActivity3D: Focuses on daily human-object interactions.
  • Online RGB-D Action (ORGBD): Living room environment actions.
  • G3D Gaming Action: Sports and gaming movements (e.g., punching, kicking).
  • Watch-n-Patch: Complex, varying sequences of office and kitchen actions.
  • KARD (Kinect Activity Recognition Dataset): 18 different action classes.

Key Insights and Ideas

  • Locality is Key: The "Aha!" moment is that motion alone isn't enough; where that motion occurs on the body provides vital context. This allows the system to distinguish between actions that look similar in motion but involve different limbs (e.g., a right-hand punch vs. a left-hand punch).
  • Radial Motion Matters: By using 3D trajectories, the system can finally "see" depth. Actions like "playing guitar" or "using a cellphone," which involve subtle movements toward the body, showed massive accuracy gains when moving from 2D to 3D.
  • Smart Filtering Beats Raw Data: Simply having more data (3D) isn't enough because it's noisy. The paper shows that filtering trajectories based on their proximity to joints and their contribution to classifier confidence is essential for performance.

Key Results

The Localized Trajectories approach consistently outperformed standard Dense Trajectories across all benchmarks:

  • MSRDailyActivity3D: Accuracy jumped from 64.4% (Standard DT) to 74.4% (2D Localized) and reached 76.3% with the 3D version.
  • G3D Dataset: Achieved 87.8% accuracy, a significant boost over the 80.1% of standard DT.
  • Watch-n-Patch: In the kitchen environment, 2D Localized Trajectories outperformed standard DT by a staggering 25.3%.
  • Specific Action Gains: For "playing guitar," accuracy climbed from 20% (Standard DT) to 70% (3D Localized), proving the value of depth-aware tracking.

Limitations and Future Work

  • Low-Motion Sensitivity: The method still struggles with actions that involve very little movement (e.g., "sitting still" or "calling on a cellphone"), a limitation inherited from the underlying trajectory-based framework.
  • Scene Flow Noise: While 3D information is helpful, scene flow estimation is currently less stable than 2D optical flow, which can degrade codebook quality if not carefully managed.
  • Future Directions: The authors plan to automate parameter selection and explore generating 3D trajectories directly from raw point cloud data to improve robustness against sensor noise.

Summary updated

Abstract

The Dense Trajectories concept is one of the most successful approaches in action recognition, suitable for scenarios involving a significant amount of motion. However, due to noise and background motion, many generated trajectories are irrelevant to the actual human activity and can potentially lead to performance degradation. In this paper, we propose Localized Trajectories as an improved version of Dense Trajectories where motion trajectories are clustered around human body joints provided by RGB-D cameras and then encoded by local Bag-of-Words. As a result, the Localized Trajectories concept provides an advanced discriminative representation of actions. Moreover, we generalize Localized Trajectories to 3D by using the depth modality. One of the main advantages of 3D Localized Trajectories is that they describe radial displacements that are perpendicular to the image plane. Extensive experiments and analysis were carried out on five different datasets.
BibTeX
@inproceedings{papadopoulos2019localized,
  author = {Konstantinos Papadopoulos and Girum G. Demisse and Enjie Ghorbel and Michel Antunes and Djamila Aouada and Björn E. Ottersten},
  title = {{Localized Trajectories for 2D and 3D Action Recognition}},
  booktitle = {Sensors},
  year = {2019},
  url = {https://openreview.net/forum?id=t1w11r8yTW},
}

Loading related content...

Figure