Hierarchical Context-aware Network for Dense Video Event Captioning

Lei Ji, Xianglin Guo, Haoyang Huang, Xilin ChenACL 2021 · OpenReview

TL;DR

This paper introduces a novel Hierarchical Context-aware Network (HCN) for dense video event captioning. HCN effectively leverages both local (neighboring frames) and global (inter-event) temporal context to generate more coherent and less redundant captions for long, untrimmed videos, outperforming previous context-agnostic methods.

What Problem Does This Paper Address?

The proliferation of video content necessitates efficient ways to understand and access information within it. Dense video event captioning aims to address this by generating descriptive captions for individual events within a long video, enabling quick skimming and search. However, existing methods often struggle with generating consistent and non-repetitive captions because they primarily focus on the visual and textual information within each event's specific clip, neglecting the broader temporal context. This can lead to issues like ambiguous object recognition (e.g., mistaking "chicken" for "pork" due to visual similarity or co-reference) and repetitive descriptions when events are visually similar. This paper tackles the challenge of incorporating both local and global temporal context to improve the coherence and quality of dense video event captions.

Methods

The core contribution is the Hierarchical Context-aware Network (HCN), designed to capture context from various temporal aspects. HCN consists of four main modules:

  1. Local Context Module (LCM): This module enhances event features by considering temporally close frames (neighboring clips). It uses a flat attention mechanism for full interaction between the source event's features and these local context features. For multi-modal inputs (video + speech text), it also fuses these modalities.
  2. Global Context Module (GCM): This module captures inter-event context by attending to previous and future events. It employs a cross-attention mechanism, with a source encoder (SEncoder) and a cross encoder (CEncoder), to selectively attend to relevant events and model temporal dependencies.
  3. Context Gating: A mechanism (using GRU gates and a gating scalar) is used to combine the context-enhanced features from LCM and GCM with the original source event features, allowing the model to dynamically weigh the importance of different contextual information.
  4. Decoder: An auto-regressive transformer model generates the caption tokens sequentially.

The model also incorporates type embeddings to distinguish between source and context features, aiding the attention mechanisms.

Datasets

The paper evaluates HCN on two benchmark datasets:

  • Youcook2: A dataset focused on instructional videos for recipe preparation.
  • ActivityNet Captions: An open-domain dataset with overlapping event proposals.

The authors follow established data partitions for both datasets and use ground-truth event proposals.

Key Insights and Ideas

  • Context is King (for Coherence): The most significant insight is that explicitly modeling both local and global temporal context is crucial for generating coherent and non-redundant dense video captions. Ignoring context leads to inconsistencies, especially in ambiguous situations.
  • Hierarchical Approach Works: A hierarchical structure that first processes local context and then global context allows for a more nuanced understanding of temporal relationships.
  • Gating for Dynamic Context Integration: The use of gating mechanisms effectively balances the influence of local and global context, as well as the source event itself, leading to better feature representation.
  • Type Embeddings Help: Distinguishing between source and context information via type embeddings aids the attention mechanisms in correctly interpreting relationships.
  • Beyond Single Clips: The paper demonstrates that moving beyond processing each event in isolation and instead considering its temporal neighborhood and its place within the larger video narrative significantly improves caption quality.

Key Results

  • Youcook2: HCN (uni-modal) outperforms the context-agnostic base model by a large margin, achieving state-of-the-art results. For multi-modal inputs, HCN achieves comparable results to other state-of-the-art methods.
  • Activitynet Captions: HCN also shows superior performance compared to baseline methods, demonstrating its effectiveness across different video domains.
  • Ablation Studies: Removing components like type embeddings or context modules leads to performance degradation, validating the importance of each part of the HCN architecture. The study also shows that both past and future contexts are beneficial, and the GRU gate is more effective than simple combinations.

Limitations and Future Work

  • Fine-grained Entity Recognition: The paper acknowledges that fine-grained object recognition from video remains a challenging problem, even with context. The model, like others, can still make mistakes in identifying very specific entities (e.g., "salt").
  • Visual Coreference Resolution: The authors suggest that visual coreference resolution could be a future direction to further improve entity consistency.
  • Long-Term Context Filtering: Future work could focus on extending the hierarchical network to better attend to long contexts and filter out ambiguous or irrelevant information.

Summary updated

BibTeX
@inproceedings{ji2021hierarchical,
  author = {Lei Ji and Xianglin Guo and Haoyang Huang and Xilin Chen},
  title = {{Hierarchical Context-aware Network for Dense Video Event Captioning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=EMSLWVlT6N},
}
Figure