Guiding the Flowing of Semantics: Interpretable Video Captioning via POS Tag

Xinyu Xiao, Lingfeng Wang, Bin Fan, Shiming Xiang, Chunhong PanEMNLP 2019 · OpenReview

Abstract

Xinyu Xiao, Lingfeng Wang, Bin Fan, Shinming Xiang, Chunhong Pan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

2019.

Introduction

Video captioning aims to automatically generate natural language descriptions for video sequences. While standard encoder-decoder models have made significant strides, they often operate as "black boxes." These models typically compress video frames into a single, dense feature vector, leading to two primary issues: semantic mixing and a lack of interpretability. In semantic mixing, distinct information types—such as objects (nouns), actions (verbs), and relationships—are blended together, making it difficult for the decoder to selectively attend to relevant details. Furthermore, it is often unclear whether a model is truly "seeing" an action or merely predicting words based on linguistic statistical patterns.

Inspired by neuroscience, which suggests the human brain activates different neural patterns for nouns and verbs, the authors propose the Adaptive Semantic Guidance Network (ASGN). This model explicitly separates semantic categories using Part-of-Speech (POS) tags to "shunt" visual information into specialized streams.

Methods: The Adaptive Semantic Guidance Network

The ASGN framework introduces a structured approach to video feature processing by integrating POS information directly into the learning pipeline.

The ASGN architecture, featuring 2D/3D CNN encoders, the POS-Aware Semantic Guider, and the dual-stream LSTM decoder for description generation and POS tag learning.

POS-Aware Semantic Guider

The core innovation is the POS-Aware Semantic Guider. During the encoding phase, the model predicts the POS tag for the upcoming word. This predicted tag acts as a controller, activating specific feature channels within the CNN representations. By applying a Hadamard product operation, the model "parses" global video features into POS-specific representations (e.g., noun-aware vs. verb-aware features). This ensures that the decoder receives information tailored to the specific grammatical role it is currently fulfilling.

Adaptive Attention and the Sentinel Gate

Not all words in a caption are grounded in visual data; functional words like "the" or "is" rely more on language modeling than visual evidence. ASGN employs an Adaptive Attention Model featuring a sentinel gate ($\beta_t$). This gate uses the instantiated noun and verb features as supervision to dynamically decide how much visual information to incorporate into the decoder versus relying on the internal state of the language model.

Optimization

The network is trained in two stages: first using standard Maximum Likelihood Estimation (MLE), followed by Reinforcement Learning (RL) to directly optimize for the CIDEr metric, ensuring the generated captions are both grammatically correct and semantically rich.

Experimental Results

The researchers evaluated ASGN on two major benchmarks: Youtube2Text (MSVD) and MSR-VTT. For training, they utilized the Polyglot toolkit to generate ground-truth POS tags across 17 universal categories. The model outperformed several state-of-the-art methods (like SA-LSTM and PickNet) across standard metrics:

  • Youtube2Text: Achieved a BLEU-4 of 0.521 and a CIDEr score of 0.803.
  • MSR-VTT: Achieved a BLEU-4 of 0.405 and a CIDEr score of 0.490.

Ablation studies confirmed that both the POS guidance and the adaptive attention mechanism were critical to performance. Human evaluations further supported these findings, with participants preferring ASGN-generated captions over baseline models in 32.9% of cases.

Interpretability and Qualitative Analysis

One of the most significant advantages of ASGN is its inherent interpretability. By tying specific network channels to POS tags, the model's "reasoning" becomes visible.

Qualitative results showing the model's ability to predict POS tags and generate captions with associated confidence scores across various video scenarios.

As shown in the qualitative examples, the model maintains high confidence when predicting key semantic anchors like nouns ("man", "couch") and verbs ("sitting"). This visualization allows researchers to verify that the model is indeed focusing on the correct visual cues for specific parts of speech, rather than relying solely on language priors.

Conclusion

The Adaptive Semantic Guidance Network demonstrates that explicitly modeling the "flow" of semantics via POS tags significantly enhances both the performance and transparency of video captioning systems. While challenges remain in grounding more abstract modifiers like adjectives and adverbs, the "shunting" mechanism provides a robust foundation for future work in multi-modal understanding and interpretable AI.

Summary updated

BibTeX
@inproceedings{xiao2019guiding,
  author = {Xinyu Xiao and Lingfeng Wang and Bin Fan and Shiming Xiang and Chunhong Pan},
  title = {{Guiding the Flowing of Semantics: Interpretable Video Captioning via POS Tag}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=DC-OrktzGjD},
}
Figure