Spatio-Temporal Dynamics and Semantic Attribute Enriched Visual Encoding for Video Captioning

Nayyer Aafaq, Naveed Akhtar, Wei Liu, Syed Zulqarnain Gilani, Ajmal Mian

Introduction

Video captioning—the task of automatically describing video clips in natural language—sits at the complex intersection of Computer Vision and Natural Language Processing. While recent research has heavily focused on developing more sophisticated language decoders, this paper argues that the primary bottleneck often lies in the visual encoder.

Standard practices, such as mean-pooling CNN features across all frames, effectively flatten the video and wash away critical temporal dynamics and fine-grained object information. This paper introduces a novel approach that prioritizes richer visual features to achieve superior captioning performance even with a relatively simple language model.

Methods: Enriched Visual Encoding (GRU-EVE)

The authors propose a visual encoding technique that produces a "visual code" enriched with both spatio-temporal dynamics and high-level semantics. The architecture, dubbed GRU-EVE, is built on three main pillars:

The GRU-EVE architecture, integrating spatio-temporal dynamics via hft and semantic embeddings.
  1. Hierarchical Short Fourier Transform (hft): Instead of simple averaging, a recursive, tree-like Fourier Transform captures how visual features change over time at multiple scales (the whole video, halves, quarters, etc.). This preserves the temporal "rhythm" and evolution of the scene for both 2D and 3D CNN features.
  2. Object Semantics and Spatial Evolution: By employing an object detector (YOLO), the model tracks specific objects along with their plurality, frequency, and velocity. This allows the system to distinguish between subtle differences, such as "a person walking" versus "people running."
  3. Action Semantics: Explicit action labels are extracted from the final output layer of a 3D CNN and intersected with the language model's vocabulary to ensure the visual features are aligned with the words the model can generate.

These enriched features are compressed through a fully connected layer and fed into a two-layer Gated Recurrent Unit (GRU) for sequence modeling.

Experimental Setup and Results

The model was evaluated on two benchmark datasets: MSVD (1,970 YouTube clips) and MSR-VTT (10,000 clips across 20 categories). GRU-EVE established new state-of-the-art performance on both benchmarks at the time of publication. On MSVD, it achieved a 35.0 METEOR score and a 71.5 ROUGE-L score. On MSR-VTT, it reached a 28.4 METEOR score, outperforming significantly more complex models.

Qualitative comparison of generated captions against ground truth and baseline models.

Qualitative analysis shows that the model is particularly adept at identifying the correct number of subjects and using precise action verbs. For instance, it can correctly identify "a group of elephants" where simpler models might only see "an elephant."

Conclusion

The success of GRU-EVE demonstrates that the secret to better captions often lies in the quality of the visual input rather than the complexity of the decoder. By capturing fine temporal details and explicit semantics, the model gains a better "common sense" understanding of the scene. Future work may involve combining these enriched encodings with more advanced language architectures like Transformers or optimizing the computational efficiency of the hierarchical transforms.

Summary updated

Abstract

Automatic generation of video captions is a fundamental challenge in computer vision. Recent techniques typically employ a combination of Convolutional Neural Networks (CNNs) and Recursive Neural Networks (RNNs) for video captioning. These methods mainly focus on tailoring sequence learning through RNNs for better caption generation, whereas off-the-shelf visual features are borrowed from CNNs. We argue that careful designing of visual features for this task is equally important, and present a visual feature encoding technique to generate semantically rich captions using Gated Recurrent Units (GRUs). Our method embeds rich temporal dynamics in visual features by hierarchically applying Short Fourier Transform to CNN features of the whole video. It additionally derives high level semantics from an object detector to enrich the representation with spatial dynamics of the detected objects. The final representation is projected to a compact space and fed to a language model. By learning a relatively simple language model comprising two GRU layers, we establish new state-of-the-art on MSVD and MSR-VTT datasets for METEOR and ROUGE_L metrics.
BibTeX
@inproceedings{aafaq2019spatio,
  author = {Nayyer Aafaq and Naveed Akhtar and Wei Liu and Syed Zulqarnain Gilani and Ajmal Mian},
  title = {{Spatio-Temporal Dynamics and Semantic Attribute Enriched Visual Encoding for Video Captioning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  doi = {10.1109/cvpr.2019.01277},
}
Figure