Deep Spatiotemporal Relation Learning With 3D Multi-Level Dense Fusion for Video Action Recognition

Junxuan Zhang, Haifeng HuIEEE Access 2019 · OpenReview

Abstract

Two-stream Convolutional Neural Network has shown a remarkable performance for video action recognition. Many recent works mainly focus on the fusion of appearance and motion information to obtain a robust spatiotemporal representation for action video. However, most of these networks are based on 2D convolution architecture and apply spatiotemporal fusion at the top layers of the network, which lacks the capability to take full advantage of the potential spatiotemporal relation in multiple levels of the network as well as capture the temporal dynamic in low-level details. In this paper, we propose a novel convolutional fusion network based on a two-stream network, called 3D Multi-Level Dense Fusion (MLDF-3D) for the deep spatiotemporal relation learning. There are mainly three merits of the proposed network: (i) rather than performing fusion only at the last convolution layer or softmax layer, the MLDF-3D performs spatiotemporal fusion at multiple levels of network, which can fully explore the potential relation of features extracted from two-stream network; (ii) we introduce dense connection in MLDF-3D to make fusion features at multiple levels directly connected; and (iii) we develop a sequence segment framework for the long-range temporal structure modeling. Our experimental results show that the MLDF-3D can effectively learn the correlation of two-stream features to obtain robust action representation, and achieve the state-of-the-art performance on HMDB51 and UCF101.

Introduction

In video action recognition, the primary objective is to accurately identify human activities, such as "brushing teeth" or "applying makeup," from video sequences. Traditionally, researchers have employed "two-stream" networks: one stream captures static appearance (spatial) while the other captures optical flow (temporal motion).

However, existing models often face two significant limitations. First, they typically perform spatiotemporal fusion only at the final layers, which can lead to a loss of fine-grained, low-level details necessary for distinguishing subtle actions. Second, while 2D-CNNs excel at image processing, they struggle with temporal modeling; conversely, 3D-CNNs are computationally intensive and difficult to train from scratch.

Proposed Method: MLDF-3D

To overcome these gaps, we introduce the 3D Multi-Level Dense Fusion (MLDF-3D) network. This novel architecture is built upon three foundational pillars:

The overall architecture of the 3D Multi-Level Dense Fusion (MLDF-3D) network showing spatial, temporal, and fusion streams.

Multi-Level Interaction (MLI)

Instead of a single fusion point at the end of the network, MLDF-3D extracts features from multiple intermediate stages of the spatial and temporal backbone networks. These features are continuously integrated into a third fusion stream, allowing the model to learn spatiotemporal relations across various levels of abstraction.

Dense Connections

Inspired by the DenseNet architecture, the fusion stream utilizes dense connections. This design choice ensures that low-level features containing local details are passed forward directly to deeper layers, preventing the network from "forgetting" fine-grained movements as the feature representation becomes more semantic.

Multi-Scale Fusion Block (MSFB)

Actions in videos can occur at different scales and distances. To handle this variability, we developed the Multi-Scale Fusion Block. This component employs 3D convolution kernels of varying sizes ($3\times1\times1$, $3\times3\times3$, and $5\times5\times5$) simultaneously to capture foreground targets across multiple scales.

The Multi-Scale Fusion Block (MSFB) showing the parallel 3D convolutional layers used to capture features at different scales.

Sequence Segment Framework (SSF)

To model long-range temporal structures, the SSF divides a video into segments and samples snippets from each. A consensus module then aggregates these samples to produce a final prediction, effectively capturing the temporal evolution of an action over time.

Experimental Results and Discussion

The MLDF-3D model was evaluated on two benchmark datasets: UCF101 (13,320 videos, 101 categories) and HMDB51 (6,766 clips, 51 categories).

Performance and Efficiency

MLDF-3D achieved state-of-the-art results, with 94.42% accuracy on UCF101 and 73.14% accuracy on HMDB51. Notably, it outperformed several 3D-CNN variants like P3D ResNet. By utilizing pre-trained 2D networks (e.g., ResNet or DenseNet trained on ImageNet) as backbones, MLDF-3D converges significantly faster than 3D-CNNs that require training from scratch.

Key Insights

  • Importance of Low-Level Fusion: Accuracy improved as the number of interaction levels increased, proving that spatiotemporal relations are valuable at every depth of the network.
  • Feature Reuse: Analysis of the feature utilization ratio confirmed that upper layers effectively reuse low-level features, justifying the use of dense connections to maintain local detail alongside global context.

Conclusion

The MLDF-3D framework demonstrates that multi-level fusion and dense connections can significantly enhance spatiotemporal relation learning for action recognition. While the model remains dependent on pre-computed optical flow and carries the complexity of a three-stream architecture, it sets a new standard for performance on major benchmarks. Future research will focus on optimizing the fusion mechanism and exploring motion extraction directly from RGB frames to reduce the computational bottleneck.

Summary updated

Abstract

Two-stream Convolutional Neural Network has shown a remarkable performance for video action recognition. Many recent works mainly focus on the fusion of appearance and motion information to obtain a robust spatiotemporal representation for action video. However, most of these networks are based on 2D convolution architecture and apply spatiotemporal fusion at the top layers of the network, which lacks the capability to take full advantage of the potential spatiotemporal relation in multiple levels of the network as well as capture the temporal dynamic in low-level details. In this paper, we propose a novel convolutional fusion network based on a two-stream network, called 3D Multi-Level Dense Fusion (MLDF-3D) for the deep spatiotemporal relation learning. There are mainly three merits of the proposed network: (i) rather than performing fusion only at the last convolution layer or softmax layer, the MLDF-3D performs spatiotemporal fusion at multiple levels of network, which can fully explore the potential relation of features extracted from two-stream network; (ii) we introduce dense connection in MLDF-3D to make fusion features at multiple levels directly connected; and (iii) we develop a sequence segment framework for the long-range temporal structure modeling. Our experimental results show that the MLDF-3D can effectively learn the correlation of two-stream features to obtain robust action representation, and achieve the state-of-the-art performance on HMDB51 and UCF101.
BibTeX
@inproceedings{zhang2019deep,
  author = {Junxuan Zhang and Haifeng Hu},
  title = {{Deep Spatiotemporal Relation Learning With 3D Multi-Level Dense Fusion for Video Action Recognition}},
  booktitle = {IEEE Access},
  year = {2019},
  url = {https://openreview.net/forum?id=jRkjPcoF3v},
}
Figure