Abstract
Human action recognition in video is challenging due to the high-dimensional nature of spatiotemporal data. While Multi-Task Learning (MTL) has been used to improve performance by sharing knowledge across related actions, traditional methods often treat entire videos as single feature vectors, losing granular temporal information. This paper proposes a novel architecture that combines 3D Convolutional Neural Networks (3D-CNNs) and Long Short-Term Memory (LSTM) networks within an enhanced MTL framework. The key innovation is a video-clip sharing mechanism that identifies and leverages similar sub-motions across different action categories. Experimental results on benchmark datasets like UCF101 and HMDB51 demonstrate significant accuracy improvements over single-task and holistic MTL approaches.
Introduction
In computer vision, recognizing human actions involves interpreting complex patterns across both space and time. Traditional Multi-Task Learning (MTL) aims to improve a model's generalization by learning related tasks simultaneously. However, most existing MTL frameworks for video recognition suffer from a "congenital defect": they represent an entire video as a single, holistic feature vector. This approach discards the rich, long-term sequential data and fails to capture the fact that different actions often share specific "sub-actions." For instance, the motion of straightening one's knees is a shared component of both "standing up" and "jumping."
To address this, we present a framework that shares knowledge at the video-clip level. By decomposing actions into smaller segments, the model can learn shared spatiotemporal primitives, leading to more robust and accurate recognition.
Proposed Methodology
The proposed system integrates deep feature extraction with a sophisticated multi-task learning framework designed for sequential data.
Feature Extraction and Sequential Modeling
The process begins by splitting each input video into 25 distinct clips. These clips are processed by a 3D Convolutional Neural Network (C3D) to extract spatiotemporal features. Unlike standard 2D CNNs, 3D CNNs utilize 3D convolution and pooling operations, allowing them to capture motion information across consecutive frames effectively.
Following feature extraction, the clip features are fed into a two-layer LSTM network. The LSTM is crucial for learning the temporal dynamics—the "story" of how one sub-motion leads into the next over the duration of the action.
Enhanced Multi-Task Learning Framework
The core of the innovation lies in the MTL layer, which decomposes the model parameters into three primary components:
- Matrix $L$: Represents latent basic tasks that are shared across all action categories.
- Matrix $S$: A task-specific matrix that determines how to combine these latent tasks for a particular action.
- Matrix $T_c$ (Video Clip Sharing Layer): This matrix uses $\ell_{2,1}$ norm regularization to automatically select and weight the most relevant video clips across different categories. This allows the model to perform robust feature selection, focusing on defining movements while ignoring "noise" clips like static backgrounds.
Experimental Results
The model was evaluated on three major benchmarks: KTH, UCF101, and HMDB51. The results consistently showed that granular clip-level sharing outperforms traditional methods.
- UCF101: The proposed method achieved an accuracy of 93.4%, representing a 4.5% improvement over the single-task C3D+LSTM baseline (88.9%).
- HMDB51: The model reached 68.9% accuracy, a 5.3% jump over the single-task version.
An ablation study confirmed the importance of the $T_c$ matrix. Removing the clip-sharing mechanism resulted in a 2% drop in accuracy on UCF101, proving that sharing detailed sub-motion information is superior to sharing holistic video features.
Conclusion
By shifting the focus from video-level to clip-level knowledge sharing, this architecture provides a more nuanced understanding of human actions. The combination of 3D-CNNs for spatial-temporal features, LSTMs for long-term dependencies, and a sparsity-inducing MTL framework allows the model to identify shared "primitives" across different tasks. Future work may explore applying this volumetric knowledge-sharing approach to other domains, such as 3D medical imaging.
Summary updated