Variational Stacked Local Attention Networks for Diverse Video Captioning

Tonmoay Deb, Akib Sadmanee, Kishor Kumar Bhaumik, Amin Ahsan Ali, M Ashraful Amin, A K M Mahbubur Rahman

TL;DR

The Variational Stacked Local Attention Network (VSLAN) introduces a novel framework for video captioning that addresses feature redundancy and deterministic output limitations. By leveraging low-rank bilinear pooling to hierarchically stack and "discount" multi-source visual features, combined with a Variational POS Encoder (VaPEn), VSLAN generates syntactically diverse and natural video descriptions. The model achieves state-of-the-art performance, outperforming existing methods on the CIDEr metric by 7.8% on MSVD and 4.5% on MSR-VTT.

What Problem Does This Paper Address?

Most current video captioning models rely on standard encoder-decoder architectures that suffer from two major limitations:

  1. Feature Redundancy and Objective Mismatch: To capture rich visual information, researchers often fuse features from multiple pre-trained models (e.g., 2D CNNs for static objects and 3D CNNs for motion). However, standard fusion methods—like simple concatenation or linear layers—ignore the hierarchical dependencies and semantic overlaps between these streams, leading to redundant information.
  2. Lack of Caption Diversity: Standard models are deterministic, aiming to generate a single "correct" caption for a video. In reality, a single video clip can be described from multiple valid perspectives and sentence structures. Generating diverse yet accurate captions remains an open challenge in the temporal domain.

Methods

VSLAN addresses these challenges through an end-to-end architecture consisting of three primary components:

  • Local Attention Network (LAN): Instead of standard linear attention, LAN utilizes low-rank bilinear pooling (via the Hadamard product) to capture fine-grained, second-order interactions within a single visual feature stream.
  • Feature Aggregation Network (FAN): Rather than fusing all visual features at a single stage, FAN blocks stack multiple feature streams sequentially. Crucially, each block learns to "discount" redundant information already captured by preceding layers, ensuring a compact and distilled visual representation.
  • Variational POS Encoder (VaPEn): To inject natural diversity, the authors introduce a stochastic latent space module based on a Variational Recurrent Neural Network (VRNN). VaPEn predicts diverse Part-of-Speech (POS) sequences conditioned on the global visual features. These POS sequences then act as a syntactic blueprint to guide the caption decoder.
Figure 1. Overview of Variational Stacked Local Attention Network (VSLAN) comprising of Local Attention Network (LAN), Feature Aggregation Network (FAN), and Variational POS Encoder (VaPEn). The final Local feature set from FAN is utilized to query caption words at the decoding stage, done by another dedicated LAN block. VaPEn is built on top of the output of each FAN block. The final output from VaPEn’s final node accounts for global POS feature, G, further added with Θn for influencing the diversity on decoder.

Shared Learning Strategy

During training, the model balances fluency and logical consistency by combining word-level Cross-Entropy (XE) loss with Reinforcement Learning (RL). Instead of relying solely on standard n-gram metrics like CIDEr for the RL reward, they utilize a textual entailment reward powered by RoBERTa to ensure that generated captions are logically inferred from the video context.

Datasets

The model was trained and evaluated on two standard video captioning benchmarks:

  • MSVD (Microsoft Research Video Description / YouTube2Text): A collection of YouTube clips with multiple parallel language descriptions.
  • MSR-VTT (MSR Video to Text): A larger, more complex dataset containing open-domain videos with a wide variety of visual content.

Key Insights and Ideas

  • Information Discounting over Fusion: Simply adding or concatenating features from different models introduces noise. Explicitly modeling what a new feature stream adds relative to what the network already knows yields much cleaner representations.
  • Syntax as a Lever for Diversity: Directly generating diverse words often leads to nonsensical sentences. By generating a diverse Part-of-Speech template first (via VaPEn), the model can explore different grammatical structures while keeping the semantic content grounded in the video.
  • Entailment over N-Grams: Standard metrics like CIDEr can reward captions that match keywords but contain critical logical errors (e.g., wrong action or object negation). Incorporating a deep language model (RoBERTa) for entailment scoring ensures the generated captions are factually consistent with the ground truth.

Key Results

VSLAN achieves significant quantitative improvements over state-of-the-art models on both datasets:

  • MSVD: The full VSLAN model achieves a CIDEr score of 98.1, outperforming its closest competitor (C-R) by 7.8%.
  • MSR-VTT: VSLAN achieves a CIDEr score of 55.8, a 4.5% improvement over previous non-audio methods.
  • Diversity Metrics: When evaluated on diversity metrics (such as mBleu-4, Div-1, and Div-2), VSLAN consistently generates more unique and varied sentence structures than standard Diverse Beam Search (DBS) and other baseline models, without sacrificing caption quality.

Limitations and Future Work

  • Modality Constraints: The authors explicitly restricted their evaluation on MSR-VTT to visual data only, ignoring audio and category metadata. They note that extending VSLAN to natively handle audio and category streams is a promising direction that would likely yield even higher performance.
  • Attention Failures: Qualitative analysis revealed occasional failure cases where the model failed to attend to the correct spatial or temporal regions due to a lack of training examples for highly specific or rare scenarios.

Summary updated

Abstract

While describing Spatio-temporal events in natural language, video captioning models mostly rely on the encoder's latent visual representation. Recent progress on the encoder-decoder model attends encoder features mainly in linear interaction with the decoder. However, growing model complexity for visual data encourages more explicit feature interaction for fine-grained information, which is currently absent in the video captioning domain. Moreover, feature aggregations methods have been used to unveil richer visual representation, either by the concatenation or using a linear layer. Though feature sets for a video semantically overlap to some extent, these approaches result in objective mismatch and feature redundancy. In addition, diversity in captions is a fundamental component of expressing one event from several meaningful perspectives, currently missing in the temporal, i.e., video captioning domain. To this end, we propose Variational Stacked Local Attention Network (VSLAN), which exploits low-rank bilinear pooling for self-attentive feature interaction and stacking multiple video feature streams in a discount fashion. Each feature stack's learned attributes contribute to our proposed diversity encoding module, followed by the decoding query stage to facilitate end-to-end diverse and natural captions without any explicit supervision on attributes. We evaluate VSLAN on MSVD and MSR-VTT datasets in terms of syntax and diversity. The CIDEr score of VSLAN outperforms current off-the-shelf methods by $7.8\%$ on MSVD and $4.5\%$ on MSR-VTT, respectively. On the same datasets, VSLAN achieves competitive results in caption diversity metrics.
BibTeX
@inproceedings{deb2022variational,
  author = {Tonmoay Deb and Akib Sadmanee and Kishor Kumar Bhaumik and Amin Ahsan Ali and M Ashraful Amin and A K M Mahbubur Rahman},
  title = {{Variational Stacked Local Attention Networks for Diverse Video Captioning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure