Spatio-Temporal Dynamics and Semantic Attribute Enriched Visual Encoding for Video Captioning

Nayyer Aafaq, Naveed Akhtar, Wei Liu, Syed Zulqarnain Gilani, Ajmal Mian

Abstract

Automatic generation of video captions is a fundamental challenge in computer vision. Recent techniques typically employ a combination of Convolutional Neural Networks (CNNs) and Recursive Neural Networks (RNNs) for video captioning. These methods mainly focus on tailoring sequence learning through RNNs for better caption generation, whereas off-the-shelf visual features are borrowed from CNNs. We argue that careful designing of visual features for this task is equally important, and present a visual feature encoding technique to generate semantically rich captions using Gated Recurrent Units (GRUs). Our method embeds rich temporal dynamics in visual features by hierarchically applying Short Fourier Transform to CNN features of the whole video. It additionally derives high level semantics from an object detector to enrich the representation with spatial dynamics of the detected objects. The final representation is projected to a compact space and fed to a language model. By learning a relatively simple language model comprising two GRU layers, we establish new state-of-the-art on MSVD and MSR-VTT datasets for METEOR and ROUGE_L metrics.

Introduction

Video captioning—the task of automatically generating a natural language description of a video—is a "bicephalic" challenge, requiring expertise in both Computer Vision (CV) and Natural Language Processing (NLP). While NLP models like LSTMs and GRUs have become highly sophisticated, the visual encoding step has often remained stagnant. Most existing methods simply "mean pool" features from a pre-trained CNN across all video frames. This lossy operation effectively flattens the video's temporal structure, discarding the fine-grained dynamics of how objects move and interact over time. This paper introduces Enriched Visual Encoding (EVE), arguing that a better visual "vocabulary" is needed before the language model even starts generation.

Methodology

The core contribution is the EVE pipeline, which consists of three main components fused into a single compact representation:

Overview of the Enriched Visual Encoding (EVE) framework.

Hierarchical Short Fourier Transform (hft)

Instead of mean pooling, the authors apply a Short Fourier Transform (SFT) to the activations of 2D (InceptionResNetV2) and 3D (C3D) CNNs. By applying this hierarchically (breaking the video into halves, then quarters), they capture the frequency-domain representation of temporal changes. This preserves the "rhythm" and dynamics of the video better than a simple average, allowing the model to distinguish between similar actions based on their rate of change.

Semantic Enrichment

The model explicitly injects high-level semantics into the visual features:

  1. Object Dynamics & Plurality: Using an object detector (YOLO), the model identifies specific objects, calculates their velocity between frames, and tracks their frequency of occurrence (plurality). This provides the language model with explicit "nouns" and spatial behavior.
  2. Action Semantics: The model extracts high-level action attributes (verbs) from the output layer of a 3D CNN.

These features are concatenated into a high-dimensional vector, compressed via a fully connected layer, and fed into a two-layer Gated Recurrent Unit (GRU) language model.

Experimental Results

The authors evaluated EVE on two gold-standard benchmarks: MSVD (Microsoft Video Description) and MSR-VTT (MSR-Video to Text). The results established new state-of-the-art (SOTA) performance at the time of publication.

Performance comparison on the MSVD dataset across multiple metrics.

On the MSVD dataset, EVE achieved a METEOR score of 35.0 and a ROUGE-L of 71.5. Ablation studies demonstrated that the hierarchical Fourier Transform (hft) alone provided a substantial boost over traditional mean pooling, validating the core hypothesis that frequency-domain temporal modeling is superior for action recognition and description.

Conclusion

This paper proves that an encoder-centric design can outperform complex decoders. By enriching the visual input with spatio-temporal dynamics and explicit semantics, the researchers achieved SOTA results using a relatively simple language model. Future work may involve integrating these enriched features into Transformer-based architectures to further push the boundaries of video understanding.

Summary updated

Abstract

Automatic generation of video captions is a fundamental challenge in computer vision. Recent techniques typically employ a combination of Convolutional Neural Networks (CNNs) and Recursive Neural Networks (RNNs) for video captioning. These methods mainly focus on tailoring sequence learning through RNNs for better caption generation, whereas off-the-shelf visual features are borrowed from CNNs. We argue that careful designing of visual features for this task is equally important, and present a visual feature encoding technique to generate semantically rich captions using Gated Recurrent Units (GRUs). Our method embeds rich temporal dynamics in visual features by hierarchically applying Short Fourier Transform to CNN features of the whole video. It additionally derives high level semantics from an object detector to enrich the representation with spatial dynamics of the detected objects. The final representation is projected to a compact space and fed to a language model. By learning a relatively simple language model comprising two GRU layers, we establish new state-of-the-art on MSVD and MSR-VTT datasets for METEOR and ROUGE_L metrics.
BibTeX
@inproceedings{aafaq2019spatio,
  author = {Nayyer Aafaq and Naveed Akhtar and Wei Liu and Syed Zulqarnain Gilani and Ajmal Mian},
  title = {{Spatio-Temporal Dynamics and Semantic Attribute Enriched Visual Encoding for Video Captioning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  doi = {10.1109/cvpr.2019.01277},
}
Figure