A 3D-CNN and LSTM Based Multi-Task Learning Architecture for Action Recognition

Xi Ouyang, Shuangjie Xu, Chaoyun Zhang, Pan Zhou, Yang Yang, Guanghui Liu, Xuelong Li

Abstract

Human action recognition in video is challenging due to the high-dimensional nature of spatiotemporal data. While Multi-Task Learning (MTL) has been used to improve performance by sharing knowledge across related actions, traditional methods often treat entire videos as single feature vectors, losing granular temporal information. This paper proposes a novel architecture that combines 3D Convolutional Neural Networks (3D-CNNs) and Long Short-Term Memory (LSTM) networks within an enhanced MTL framework. The key innovation is a video-clip sharing mechanism that identifies and leverages similar sub-motions across different action categories. Experimental results on benchmark datasets like UCF101 and HMDB51 demonstrate significant accuracy improvements over single-task and holistic MTL approaches.

Introduction

In computer vision, recognizing human actions involves interpreting complex patterns across both space and time. Traditional Multi-Task Learning (MTL) aims to improve a model's generalization by learning related tasks simultaneously. However, most existing MTL frameworks for video recognition suffer from a "congenital defect": they represent an entire video as a single, holistic feature vector. This approach discards the rich, long-term sequential data and fails to capture the fact that different actions often share specific "sub-actions." For instance, the motion of straightening one's knees is a shared component of both "standing up" and "jumping."

To address this, we present a framework that shares knowledge at the video-clip level. By decomposing actions into smaller segments, the model can learn shared spatiotemporal primitives, leading to more robust and accurate recognition.

Proposed Methodology

The proposed system integrates deep feature extraction with a sophisticated multi-task learning framework designed for sequential data.

The complete architecture of the proposed 3D-CNN and LSTM based Multi-Task Learning framework.

Feature Extraction and Sequential Modeling

The process begins by splitting each input video into 25 distinct clips. These clips are processed by a 3D Convolutional Neural Network (C3D) to extract spatiotemporal features. Unlike standard 2D CNNs, 3D CNNs utilize 3D convolution and pooling operations, allowing them to capture motion information across consecutive frames effectively.

The C3D architecture used for extracting spatiotemporal features from video clips.

Following feature extraction, the clip features are fed into a two-layer LSTM network. The LSTM is crucial for learning the temporal dynamics—the "story" of how one sub-motion leads into the next over the duration of the action.

Enhanced Multi-Task Learning Framework

The core of the innovation lies in the MTL layer, which decomposes the model parameters into three primary components:

  1. Matrix $L$: Represents latent basic tasks that are shared across all action categories.
  2. Matrix $S$: A task-specific matrix that determines how to combine these latent tasks for a particular action.
  3. Matrix $T_c$ (Video Clip Sharing Layer): This matrix uses $\ell_{2,1}$ norm regularization to automatically select and weight the most relevant video clips across different categories. This allows the model to perform robust feature selection, focusing on defining movements while ignoring "noise" clips like static backgrounds.

Experimental Results

The model was evaluated on three major benchmarks: KTH, UCF101, and HMDB51. The results consistently showed that granular clip-level sharing outperforms traditional methods.

  • UCF101: The proposed method achieved an accuracy of 93.4%, representing a 4.5% improvement over the single-task C3D+LSTM baseline (88.9%).
  • HMDB51: The model reached 68.9% accuracy, a 5.3% jump over the single-task version.

An ablation study confirmed the importance of the $T_c$ matrix. Removing the clip-sharing mechanism resulted in a 2% drop in accuracy on UCF101, proving that sharing detailed sub-motion information is superior to sharing holistic video features.

Conclusion

By shifting the focus from video-level to clip-level knowledge sharing, this architecture provides a more nuanced understanding of human actions. The combination of 3D-CNNs for spatial-temporal features, LSTMs for long-term dependencies, and a sparsity-inducing MTL framework allows the model to identify shared "primitives" across different tasks. Future work may explore applying this volumetric knowledge-sharing approach to other domains, such as 3D medical imaging.

Summary updated

Abstract

Multi-task learning (MTL) is a machine learning method to share knowledge for multiple related machine learning tasks via learning those tasks jointly. It has been shown to be capable of effectively improving the generalization capability of each single task (learning just one task at a time). In this paper, we propose a novel MTL architecture that first combines 3D convolutional neural networks (3D CNN) plus the long short-term memory (LSTM) networks together with the MTL mechanism, tailored to information sharing of video inputs. We split each video into several clips and apply the hybrid deep model of 3D CNN and LSTM to extract the sequential features of those video clips. Therefore, our MTL model can share visual knowledge based on those video-clip features among different categories more efficiently. We evaluate our method on three popular public action recognition video datasets. The experimental results show that our novel MTL method can efficiently share detailed information in video clips among multiple action categories and outperforms other multi-task methods.
BibTeX
@article{ouyang2019d,
  author = {Xi Ouyang and Shuangjie Xu and Chaoyun Zhang and Pan Zhou and Yang Yang and Guanghui Liu and Xuelong Li},
  title = {{A 3D-CNN and LSTM Based Multi-Task Learning Architecture for Action Recognition}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2906654},
}
Figure