Abstract
Recognizing human actions in video requires capturing both spatial appearance and temporal dynamics. While Trajectory-pooled Deep-learning Descriptors (TDD) have shown promise by pooling CNN features along dense trajectories, they typically rely on average pooling, which discards the temporal order of features. This paper introduces Evolution-Preserving Trajectory (EPT) descriptors. By applying rank pooling to individual trajectories, EPT descriptors encode the temporal evolution of features, significantly outperforming standard TDD on major benchmarks like Hollywood2 and UCF101. Our results demonstrate that the "how" of motion—the specific sequence of changes—is a highly discriminative cue for action recognition.
Introduction: Beyond Average Pooling
Human action recognition is challenged by background clutter, camera motion, and the inherent complexity of human movement. A significant milestone in this field was the development of Dense Trajectory Descriptors (DTD), which track local points through video sequences. More recently, Trajectory-pooled Deep-learning Descriptors (TDD) improved upon this by extracting features from Convolutional Neural Networks (CNNs) along these trajectories.
However, a fundamental limitation of TDD is its reliance on "average pooling." By averaging deep features across a 15-frame trajectory, the method effectively "smears" temporal information, ignoring the sequence in which appearance or motion evolves. As shown in the comparison below, this loss of order makes it difficult to distinguish between actions that share similar average appearances but differ in their temporal progression.
Methodology: The EPT Pipeline
To address the limitations of average pooling, we propose the Evolution-Preserving Trajectory (EPT) descriptor. The pipeline consists of four primary stages:
- Trajectory Extraction: We utilize "Improved Dense Trajectories" to identify and track points of interest across short temporal windows (typically 15 frames).
- Deep Feature Mapping: Video frames are processed through a two-stream CNN architecture. One stream captures spatial/appearance information, while the other captures temporal/optical flow information.
- Rank Pooling: Instead of averaging features, we apply Rank Pooling. This technique treats the sequence of features as a ranking problem, seeking a vector that captures the "direction" and "evolution" of change over time. To maintain computational efficiency across hundreds of thousands of trajectories, we employ Approximate Rank Pooling, which reduces the optimization to a fast linear operation.
- In-Voxel Normalization: We introduce a normalization step that ensures every feature channel contributes equitably before pooling, preventing high-magnitude channels from dominating the descriptor.
Key Insights and Results
Our evaluation on the Hollywood2 and UCF101 datasets yielded several critical insights:
- Local Evolution is Highly Discriminative: While previous methods applied rank pooling to entire video clips, our work proves that temporal evolution is powerful even at the micro-level of a 15-frame trajectory.
- Spatial Stream Gains: Interestingly, the EPT method provided a more substantial boost to the spatial (appearance) stream than the temporal stream. On UCF101, spatial EPT descriptors achieved 84.1% accuracy, a 6.6% improvement over spatial TDD (77.5%). This suggests that the way an object's appearance changes during motion is an underutilized but potent cue.
- State-of-the-Art Performance: The combined two-stream EPT model reached 88.7% on UCF101. On Hollywood2, the method achieved a mean Average Precision (mAP) of 78.6%, setting a new record at the time of publication.
- Normalization Matters: We found that "in-voxel" normalization (normalizing across channels for a specific point in space-time) was significantly more effective than standard "in-channel" normalization, ensuring the rank pooling process captures meaningful evolution across all features.
Conclusion and Future Work
EPT descriptors represent a significant advancement in capturing the nuances of human motion by preserving the temporal order of deep features. While highly effective for dynamic actions, the method faces challenges in static scenes where little change occurs. Future research will focus on developing hybrid descriptors that can seamlessly integrate the strengths of both static appearance (average pooling) and dynamic evolution (rank pooling) to provide a more robust representation of human activity in video.
Summary updated