Visual-aware Attention Dual-stream Decoder for Video Captioning

Zhixin Sun, Xian Zhong, Shuqin Chen, Lin Li, Zhong LuoICME 2021

Abstract

Video captioning is a challenging task that captures different visual parts and describes them in sentences, for it requires visual and linguistic coherence. The attention mechanism in the current video captioning method learns to assign weight to each frame, promoting the decoder dynamically. This may not explicitly model the correlation and the temporal coherence of the visual features extracted in the sequence frames. To generate semantically coherent sentences, we propose a new Visual-aware Attention (VA) model, which concatenates dynamic changes of temporal sequence frames with the words at the previous moment, as the input of attention mechanism to extract sequence features. In addition, the prevalent approaches widely use the teacher-forcing (TF) learning during training, where the next token is generated conditioned on the previous ground-truth tokens. The semantic information in the previously generated tokens is lost. Therefore, we design a self-forcing (SF) stream that takes the semantic information in the probability distribution of the previous token as input to enhance the current token. The Dual-stream Decoder (DD) architecture unifies the TF and SF streams, generating sentences to promote the annotated captioning for both streams. Meanwhile, with the Dual-stream Decoder utilized, the exposure bias problem is alleviated, caused by the discrepancy between the training and testing in the TF learning. The effectiveness of the proposed Visual-aware Attention Dual-stream Decoder (VADD) is demonstrated through the result of experimental studies on Microsoft video description (MSVD) corpus and MSR-Video to text (MSR-VTT) datasets.

Introduction

Video captioning—automatically describing video content in natural language—is significantly harder than image captioning because it requires both visual and linguistic coherence over time. Current models face two primary hurdles:

  1. Static Attention: Standard attention mechanisms assign weights to video frames independently at each time step, often failing to capture the temporal correlation and dynamic transitions between frames as a sentence progresses.
  2. Exposure Bias and Semantic Loss: Most models are trained using "Teacher Forcing" (TF), where the model predicts the next word based on the ground-truth previous word. During testing, the model relies on its own previous predictions, leading to exposure bias. Furthermore, by only using the single "best" word from the previous step, models lose the rich semantic information contained in the full probability distribution.

Methodology

The authors propose the VADD framework, which integrates a novel attention mechanism with a dual-stream decoding architecture.

The overall architecture of the VADD model, showing the integration of the Visual-aware Attention (VA) and the Dual-stream Decoder.

Visual-aware Attention (VA)

Unlike traditional attention that looks at frames in isolation, the VA module uses a shared LSTM to track the "history" of visual features. It concatenates the dynamic changes of temporal sequence frames with the hidden state from the previous moment. This allows the model to "remember" what it saw and described previously, ensuring the next attended feature is temporally consistent with the narrative.

Dual-stream Decoder (DD)

To solve exposure bias and semantic loss, the decoder runs two parallel streams during training:

  • Teacher-forcing (TF) Stream: The standard approach using ground-truth tokens.
  • Self-forcing (SF) Stream: This stream takes the probability distribution of the previous token as input.
A comparison between the Teacher-forcing stream (using one-hot ground truth) and the Self-forcing stream (using probability distributions).

By utilizing the entire probability distribution, the SF stream captures semantic nuances—such as understanding that a word could have multiple meanings or synonyms—that are lost when only the top prediction is used. The model is trained using a mixed training loss that balances both streams, typically favoring the TF stream with a weight of $\lambda=0.8$. During inference, the outputs are fused using a trade-off parameter ($\gamma$) to generate the final caption.

Experiments and Results

The VADD model was evaluated on two gold-standard benchmarks: MSVD (Microsoft Video Description) and MSR-VTT (MSR-Video to Text).

Key Results

The VADD model achieved state-of-the-art or competitive results across multiple metrics:

  • MSR-VTT: Achieved a BLEU-4 of 42.4 and a CIDEr of 49.7, outperforming recent models like SGN and SAAT.
  • MSVD: Achieved a CIDEr score of 91.5, a significant improvement over the baseline (87.9).
  • Qualitative Improvements: In head-to-head comparisons, VADD was better at identifying specific objects (e.g., "motorcycle" instead of "bike") and maintaining contextual logic, such as correctly identifying "cartoon characters" rather than a generic "group of kids."

Conclusion

The VADD framework effectively addresses the challenges of temporal coherence and exposure bias in video captioning. By combining Visual-aware Attention with a Dual-stream Decoder, the model generates more semantically rich and logically consistent descriptions. While the VA mechanism showed slightly less improvement on the smaller MSVD dataset due to its shorter sentences, the overall dual-stream approach provides a robust regularizer for video-to-text tasks.

Summary updated

Abstract

Video captioning is a challenging task that captures different visual parts and describes them in sentences, for it requires visual and linguistic coherence. The attention mechanism in the current video captioning method learns to assign weight to each frame, promoting the decoder dynamically. This may not explicitly model the correlation and the temporal coherence of the visual features extracted in the sequence frames.To generate semantically coherent sentences, we propose a new Visual-aware Attention (VA) model, which concatenates dynamic changes of temporal sequence frames with the words at the previous moment, as the input of attention mechanism to extract sequence features.In addition, the prevalent approaches widely use the teacher-forcing (TF) learning during training, where the next token is generated conditioned on the previous ground-truth tokens. The semantic information in the previously generated tokens is lost. Therefore, we design a self-forcing (SF) stream that takes the semantic information in the probability distribution of the previous token as input to enhance the current token.The Dual-stream Decoder (DD) architecture unifies the TF and SF streams, generating sentences to promote the annotated captioning for both streams.Meanwhile, with the Dual-stream Decoder utilized, the exposure bias problem is alleviated, caused by the discrepancy between the training and testing in the TF learning.The effectiveness of the proposed Visual-aware Attention Dual-stream Decoder (VADD) is demonstrated through the result of experimental studies on Microsoft video description (MSVD) corpus and MSR-Video to text (MSR-VTT) datasets.
BibTeX
@inproceedings{sun2021visual,
  author = {Zhixin Sun and Xian Zhong and Shuqin Chen and Lin Li and Zhong Luo},
  title = {{Visual-aware Attention Dual-stream Decoder for Video Captioning}},
  booktitle = {ICME},
  year = {2021},
}
Figure