Context-LSTM: a robust classifier for video detection on UCF101

Dengshan Li, Rujing WangarXiv 2022 · DOI · arXiv

Abstract

Action recognition in video sequences is a computationally intensive task requiring the simultaneous modeling of spatial and temporal features. Traditional architectures often suffer from information transmission loss, where fine-grained features are degraded across deep layers, leading to inefficient training and reduced accuracy. This paper introduces Context-LSTM, a novel classifier architecture that utilizes a hierarchical cascade of bi-directional LSTM blocks integrated with intermediate pooling and normalization. Evaluated on the UCF101 dataset, the proposed model achieves a state-of-the-art 92.2% top-1 accuracy while reducing training time by approximately 41% and lowering GPU memory requirements by 2GB compared to standard CNN-LSTM baselines.

Introduction

Video action recognition is fundamentally more complex than static image classification due to the added temporal dimension. Standard approaches, such as C3D or Two-Stream networks, are often resource-heavy and computationally expensive. A primary bottleneck identified in these systems is the loss of information during transmission through multiple layers of convolutions and activations. When Long Short-Term Memory (LSTM) networks are used for temporal modeling, this loss hinders the capture of long-term dependencies. The Context-LSTM approach addresses this by optimizing the flow of temporal information through a more efficient, interconnected structure.

Methodology

The proposed architecture follows a two-stage pipeline: a spatial feature extractor followed by a specialized temporal classifier.

Spatial Backbone

A pre-trained ResNet-50 serves as the backbone, extracting high-level semantic tensors from raw input frames. This stage acts as the "eyes" of the system, identifying spatial features within individual frames.

The overall architecture of the video recognition system, showing the flow from input frames through the ResNet-50 backbone to the Context-LSTM classifier.

Context-LSTM Classifier

The core innovation lies in the Context-LSTM classifier. Rather than employing a single large LSTM layer, the model uses a hierarchical cascade of bi-directional LSTM blocks. Key features include:

  • State Interconnection: Unlike standard stacked LSTMs, Context-LSTM explicitly connects cell state flow ($C_t$) and hidden output flow ($h_t$) between blocks to preserve temporal context.
  • Intermediate Operations: Between LSTM blocks, the architecture inserts ReLU activations, Batch Normalization (BN), and MaxPooling. While MaxPooling is conventional in CNNs, its application here between temporal blocks helps reduce dimensionality and noise, which is critical for efficiency.
  • Bi-directional Processing: Sequences are processed in both forward and backward directions to strengthen temporal connectivity.
Detailed schematic of the Context-LSTM structure, highlighting the cascading LSTM blocks and the intermediate ReLU, BN, and MaxPooling layers.

Experimental Results

The model was rigorously tested on the UCF101 dataset, comprising 13,320 videos across 101 categories. To ensure robustness, the authors evaluated the model on the entire validation set.

Performance and Efficiency

The Context-LSTM demonstrated superior performance across several metrics compared to a traditional ResNet-50 + LSTM baseline:

  • Accuracy: Achieved 92.2% accuracy, outperforming the baseline (90.7%) and other models like SMART (74.6%).
  • Training Speed: Training time for 300 epochs dropped from 93 hours to 55 hours, representing a ~41% improvement. The authors suggest that better utilization of temporal information allows for faster convergence.
  • Memory Footprint: GPU memory usage was reduced from 5.5 GB to 3.6 GB, making the model more accessible for mid-range hardware.

Ablation Insights

Ablation studies confirmed that MaxPooling is vital for efficiency; removing it caused accuracy to drop and GPU usage to spike. Furthermore, the specific sequence of ReLU-then-BN was found to be more effective for training stability and final accuracy than the reverse order.

Conclusion

Context-LSTM provides a streamlined and effective solution for video action recognition by treating temporal processing with the same structural rigor as spatial processing. By incorporating intermediate pooling and normalization within a cascaded LSTM framework, the model achieves significant gains in both accuracy and computational efficiency. Future research will focus on validating this architecture on larger-scale benchmarks like Kinetics-400 and exploring its applicability to other sequence-based tasks such as speech recognition.

Summary updated

BibTeX
@misc{li2022context,
  author = {Dengshan Li and Rujing Wang},
  title = {{Context-LSTM: a robust classifier for video detection on UCF101}},
  year = {2022},
  eprint = {2203.06610},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.06610},
}
Figure