Attention-Based Convolutional LSTM for Describing Video

Zhongyu Liu, Chen Tian, Enjie Ding, Ya‐Feng Liu, Wanli YuIEEE Access 2020 · DOI

Abstract

Video description technique has been widely used in the computer community for many applications. The typical approaches are mainly based on the encode-decode framework: the fixed-length video representation vectors are extracted by the encoder using the upper layer output of pre-trained convolutional neural networks (CNNs); The decoder uses the recurrent neural networks to generate a textual sentence. However, the upper layers of convolutional neural networks contain low-resolution, semantically strong, while the lower layers contain high-resolution, semantically weak features. In the existing method, the multi-scale information of CNNs is hardly considered to be used in the video description. Ignoring this information will lead to the problem that the video description is not detailed and comprehensive. This paper applies the hierarchical convolutional long short-term memory (ConvLSTM) in the encode-decode framework to conduct feature extraction of the upper and lower layers in CNNs. Moreover, multiple network structures are designed to explore the Spatio-temporal feature extraction performance of ConvLSTM, which can approach the optimal accuracy in the three-layer ConvLSTM. In order to efficiently improve the language quality of video description, the attention mechanism focuses on visual feature output by ConvLSTM. The extensive experimental results demonstrate that the proposed method outperforms the existing approaches.

Introduction

Video description is a challenging task at the intersection of computer vision and natural language processing, requiring a model to understand temporal visual sequences and translate them into coherent natural language. The prevailing paradigm utilizes an encoder-decoder architecture where a pre-trained Convolutional Neural Network (CNN) extracts visual features, and a Recurrent Neural Network (RNN) decodes these features into a sentence.

A significant limitation of traditional models is their reliance on the final output layer of the CNN encoder. While deep layers provide strong semantic information (identifying objects), they suffer from low spatial resolution. Conversely, earlier layers retain high-resolution details that are often discarded. This paper introduces the Attention-Based Hierarchical Convolutional LSTM (AttHCLSTM) to leverage these multi-scale features for more detailed video captioning.

The AttHCLSTM Architecture

The proposed framework addresses the loss of spatial information by integrating hierarchical feature extraction with convolutional memory units.

The proposed AttHCLSTM architecture, illustrating the flow from multi-scale CNN feature extraction through a hierarchical ConvLSTM encoder to an attention-based decoder.

Hierarchical ConvLSTM (hConvLSTM)

Unlike standard LSTMs that utilize fully connected layers—thereby flattening spatial data—the ConvLSTM incorporates convolutional operations within its internal gates. This design preserves the 2D spatial structure of video frames, which is crucial for capturing fine-grained visual details over time.

Multi-Scale Feature Fusion

The encoder employs a three-layer hierarchy to progressively integrate visual information:

  • Layer 1: Processes high-resolution, low-level features from early CNN layers.
  • Layer 2: Fuses the output of the first layer with mid-level CNN features.
  • Layer 3: Combines the previous representations with high-level semantic features and 3D CNN features (e.g., from 3D ResNet-152) to capture complex motion dynamics.

Attention-Based Decoder

To translate these rich spatio-temporal features into text, the decoder employs a soft attention mechanism. This allows the model to dynamically "focus" on specific parts of the visual feature set generated by the ConvLSTM stack that are most relevant to the word currently being generated.

Experimental Evaluation

The AttHCLSTM model was evaluated on two major benchmarks: the Microsoft Video Description (MSVD) dataset and the MSR-VTT dataset.

Performance Metrics

The model demonstrated superior performance across standard evaluation metrics, including BLEU@4, METEOR, and CIDEr. On the MSVD dataset, the model achieved a BLEU@4 score of 54.1%, significantly outperforming baselines like S2VT and SA-LSTM. On the larger MSR-VTT dataset, it reached a BLEU@4 of 45.7%.

Human Evaluation

Beyond automated metrics, human evaluation was conducted to assess the descriptive quality of the generated captions. The AttHCLSTM model scored significantly higher in the "amount of detail" (M4 metric), confirming that the hierarchical integration of multi-scale features leads to more comprehensive and human-like storytelling.

Conclusion

The AttHCLSTM framework demonstrates that lower-layer CNN features are vital for generating detailed video descriptions. By maintaining spatial resolution through ConvLSTMs and fusing multi-scale information, the model achieves state-of-the-art results. Future research will focus on optimizing the computational efficiency of the hierarchical structure and exploring multi-modal inputs, such as audio, to further enhance description accuracy.

Summary updated

Abstract

Video description technique has been widely used in the computer community for many applications. The typical approaches are mainly based on the encode-decode framework: the fixed-length video representation vectors are extracted by the encoder using the upper layer output of pre-trained convolutional neural networks (CNNs); The decoder uses the recurrent neural networks to generate a textual sentence. However, the upper layers of convolutional neural networks contain low-resolution, semantically strong, while the lower layers contain high-resolution, semantically weak features. In the existing method, the multi-scale information of CNNs is hardly considered to be used in the video description. Ignoring this information will lead to the problem that the video description is not detailed and comprehensive. This paper applies the hierarchical convolutional long short-term memory (ConvLSTM) in the encode-decode framework to conduct feature extraction of the upper and lower layers in CNNs. Moreover, multiple network structures are designed to explore the Spatio-temporal feature extraction performance of ConvLSTM, which can approach the optimal accuracy in the three-layer ConvLSTM. In order to efficiently improve the language quality of video description, the attention mechanism focuses on visual feature output by ConvLSTM. The extensive experimental results demonstrate that the proposed method outperforms the existing approaches.
BibTeX
@inproceedings{liu2020attention,
  author = {Zhongyu Liu and Chen Tian and Enjie Ding and Ya‐Feng Liu and Wanli Yu},
  title = {{Attention-Based Convolutional LSTM for Describing Video}},
  booktitle = {IEEE Access},
  year = {2020},
  doi = {10.1109/access.2020.3010872},
}
Figure