Abstract
Sharing knowledge for multiple related machine learning tasks is an effective strategy to improve the generalization performance. In this paper, we investigate knowledge sharing across categories for action recognition in videos. The motivation is that many action categories are related, where common motion pattern are shared among them (e.g. diving and high jump share the jump motion). We propose a new multi-task learning method to learn latent tasks shared across categories, and reconstruct a classifier for each category from these latent tasks. Compared to previous methods, our approach has two advantages: (1) The learned latent tasks correspond to basic motion patterns instead of full actions, thus enhancing discrimination power of the classifiers. (2) Categories are selected to share information with a sparsity regularizer, avoiding falsely forcing all categories to share knowledge. Experimental results on multiple public data sets show that the proposed approach can effectively transfer knowledge between different action categories to improve the performance of conventional single task learning methods.
Introduction
Traditional action recognition systems typically employ Single Task Learning (STL), treating each action category—such as "diving" or "high jump"—as an independent classification problem. However, this approach ignores the inherent semantic and physical relationships between actions. For instance, both a diver and a high jumper execute a "jump" motion. Multi-Task Learning (MTL) aims to exploit these relationships by sharing knowledge across tasks.
Previous MTL approaches in this domain faced two primary challenges. First, "holistic over-sharing" often occurred, where sharing complex, high-level action representations made it difficult for models to distinguish between subtly different categories. Second, many models assumed all tasks were related, leading to "negative transfer" when unrelated actions (e.g., "playing violin" and "playing soccer") were forced to share information. This paper addresses these issues by decomposing actions into granular "latent tasks" and employing selective sharing.
Methodology: Latent Task Decomposition
The proposed framework represents the classifiers for all action categories as a matrix $W$. This matrix is decomposed into the product of two matrices: $W = LS$.
- Latent Tasks ($L$): This matrix acts as a dictionary of basic, shared motion patterns. By applying an $\ell_1$ norm regularizer, the model ensures these latent tasks are sparse and focused on specific feature patterns rather than holistic actions.
- Combination Weights ($S$): This matrix determines how each action category is reconstructed from the latent tasks. A sparsity constraint on $S$ ensures selective sharing, meaning only truly related categories will utilize the same latent tasks.
The optimization problem is non-convex and is solved using an Accelerated Proximal Gradient (APG) method, which iteratively updates $L$ and $S$ until convergence.
Experimental Evaluation
The model was evaluated on two benchmark datasets: UCF50, containing 50 categories with over 6,600 videos, and Olympic Sports, containing 16 categories focused on athletic movements.
Key Results
The proposed method demonstrated significant improvements over baseline STL and existing MTL techniques. On the UCF50 dataset, the approach achieved a state-of-the-art accuracy of 80.2%, a notable increase from the previous benchmark of 75.7%.
The benefits of latent task sharing are most pronounced when training data is limited. In experiments with only 25% of the UCF50 training data, the method outperformed STL by 9.0%. On the Olympic Sports dataset, the model achieved a Mean Average Precision (mAP) of 78.3%, surpassing other contemporary methods.
Discussion and Conclusion
The success of this framework highlights that granularity is essential for effective knowledge transfer in action recognition. By sharing "basic motion patterns" rather than full actions, the model maintains high discriminative power. Furthermore, the dual-sparsity approach allows for the automatic discovery of relationships between categories without manual labeling. While the current alternating optimization for the non-convex objective works efficiently, future work could explore convex formulations to provide stronger global optimality guarantees.
Summary updated