Evolution-Preserving Dense Trajectory Descriptors

Yang Wang, Vinh Cao Trần, Minh HoaiarXiv 2017 · DOI · arXiv

Abstract

Recognizing human actions in video requires capturing both spatial appearance and temporal dynamics. While Trajectory-pooled Deep-learning Descriptors (TDD) have shown promise by pooling CNN features along dense trajectories, they typically rely on average pooling, which discards the temporal order of features. This paper introduces Evolution-Preserving Trajectory (EPT) descriptors. By applying rank pooling to individual trajectories, EPT descriptors encode the temporal evolution of features, significantly outperforming standard TDD on major benchmarks like Hollywood2 and UCF101. Our results demonstrate that the "how" of motion—the specific sequence of changes—is a highly discriminative cue for action recognition.

Introduction: Beyond Average Pooling

Human action recognition is challenged by background clutter, camera motion, and the inherent complexity of human movement. A significant milestone in this field was the development of Dense Trajectory Descriptors (DTD), which track local points through video sequences. More recently, Trajectory-pooled Deep-learning Descriptors (TDD) improved upon this by extracting features from Convolutional Neural Networks (CNNs) along these trajectories.

However, a fundamental limitation of TDD is its reliance on "average pooling." By averaging deep features across a 15-frame trajectory, the method effectively "smears" temporal information, ignoring the sequence in which appearance or motion evolves. As shown in the comparison below, this loss of order makes it difficult to distinguish between actions that share similar average appearances but differ in their temporal progression.

Comparison of TDD (average pooling) and EPT (evolution-preserving pooling) across video frames. TDD collapses temporal information into a single average, while EPT captures the directional change.

Methodology: The EPT Pipeline

To address the limitations of average pooling, we propose the Evolution-Preserving Trajectory (EPT) descriptor. The pipeline consists of four primary stages:

  1. Trajectory Extraction: We utilize "Improved Dense Trajectories" to identify and track points of interest across short temporal windows (typically 15 frames).
  2. Deep Feature Mapping: Video frames are processed through a two-stream CNN architecture. One stream captures spatial/appearance information, while the other captures temporal/optical flow information.
  3. Rank Pooling: Instead of averaging features, we apply Rank Pooling. This technique treats the sequence of features as a ranking problem, seeking a vector that captures the "direction" and "evolution" of change over time. To maintain computational efficiency across hundreds of thousands of trajectories, we employ Approximate Rank Pooling, which reduces the optimization to a fast linear operation.
  4. In-Voxel Normalization: We introduce a normalization step that ensures every feature channel contributes equitably before pooling, preventing high-magnitude channels from dominating the descriptor.
The EPT pipeline: Dense trajectories are mapped to feature maps, and descriptors are generated via rank pooling (represented by the gradient symbol ∇) rather than simple summation (Σ).

Key Insights and Results

Our evaluation on the Hollywood2 and UCF101 datasets yielded several critical insights:

  • Local Evolution is Highly Discriminative: While previous methods applied rank pooling to entire video clips, our work proves that temporal evolution is powerful even at the micro-level of a 15-frame trajectory.
  • Spatial Stream Gains: Interestingly, the EPT method provided a more substantial boost to the spatial (appearance) stream than the temporal stream. On UCF101, spatial EPT descriptors achieved 84.1% accuracy, a 6.6% improvement over spatial TDD (77.5%). This suggests that the way an object's appearance changes during motion is an underutilized but potent cue.
  • State-of-the-Art Performance: The combined two-stream EPT model reached 88.7% on UCF101. On Hollywood2, the method achieved a mean Average Precision (mAP) of 78.6%, setting a new record at the time of publication.
  • Normalization Matters: We found that "in-voxel" normalization (normalizing across channels for a specific point in space-time) was significantly more effective than standard "in-channel" normalization, ensuring the rank pooling process captures meaningful evolution across all features.

Conclusion and Future Work

EPT descriptors represent a significant advancement in capturing the nuances of human motion by preserving the temporal order of deep features. While highly effective for dynamic actions, the method faces challenges in static scenes where little change occurs. Future research will focus on developing hybrid descriptors that can seamlessly integrate the strengths of both static appearance (average pooling) and dynamic evolution (rank pooling) to provide a more robust representation of human activity in video.

Summary updated

Abstract

Recently Trajectory-pooled Deep-learning Descriptors were shown to achieve state-of-the-art human action recognition results on a number of datasets. This paper improves their performance by applying rank pooling to each trajectory, encoding the temporal evolution of deep learning features computed along the trajectory. This leads to Evolution-Preserving Trajectory (EPT) descriptors, a novel type of video descriptor that significantly outperforms Trajectory-pooled Deep-learning Descriptors. EPT descriptors are defined based on dense trajectories, and they provide complimentary benefits to video descriptors that are not based on trajectories. In particular, we show that the combination of EPT descriptors and VideoDarwin leads to state-of-the-art performance on Hollywood2 and UCF101 datasets.
BibTeX
@misc{wang2017evolution,
  author = {Yang Wang and Vinh Cao Trần and Minh Hoai},
  title = {{Evolution-Preserving Dense Trajectory Descriptors}},
  year = {2017},
  eprint = {1702.04037},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1702.04037},
}
Figure