Incorporating Background Knowledge into Video Description Generation

Spencer Whitehead, Heng Ji, Mohit Bansal, Shih-Fu Chang, Clare R. VossEMNLP 2018 · OpenReview

Abstract: Whitehead et al. (2018) present a novel framework for generating knowledge-rich video descriptions by leveraging external background knowledge. While standard models excel at describing actions, they often fail to identify specific entities and context. The proposed Knowledge-aware Video Description (KaVD) network addresses this by retrieving topically related news articles to ground descriptions in real-world facts.

Introduction: The Contextual Gap in Video Captioning

Standard video captioning models are remarkably proficient at describing general actions but often fail to identify specific actors and the broader context of an event. For instance, a model might describe a political broadcast as "a man is talking," missing the critical "who, where, and why" that define the video's significance.

A video frame of Major General S. B. Moyo, illustrating the challenge of identifying specific actors without external knowledge.

The authors identify two primary challenges:

  1. Visual Insufficiency: Raw pixels often lack the information needed to identify specific named entities or the underlying causes of complex events.
  2. Contextual Gap: Recognizing a "protest" is insufficient if the model cannot distinguish between a climate rally in London and an election-related demonstration in Kenya.

The Knowledge-aware Video Description (KaVD) Framework

To bridge this gap, the authors propose the KaVD framework, which integrates external information into the generation process through three main stages:

1. Knowledge Retrieval & Extraction

Using video metadata—such as tags and upload dates—the system retrieves topically related news articles. Information Extraction (IE) tools then identify named entities (e.g., "Carles Puigdemont") and event types (e.g., "Demonstration") within these texts.

Example of the KaVD framework's input: a video of a Catalan independence demonstration with its metadata and retrieved topically related news documents.

2. The Knowledge Gate

A novel gating mechanism in the decoder uses a one-hot vector representing the extracted entity and event types to "bias" the model. For example, detecting an "Attack" event encourages the decoder to prioritize terms like "bombing" or "explosion" over generic descriptors.

3. Entity Pointer Network

Because specific names appear infrequently in training data, the model employs a "soft switch" pointer network. This allows the system to choose between generating a common word from a fixed vocabulary or "pointing" to and copying a specific entity directly from the retrieved news documents.

Datasets and Evaluation

The researchers introduced a new News Video Dataset consisting of 2,883 videos from AFP’s YouTube channel. These descriptions are "knowledge-rich," averaging 2.1 named entities per sentence—a stark contrast to existing benchmarks like MSVD or MSR-VTT, which contain almost zero named entities.

To measure performance beyond simple fluency, the authors proposed a Knowledge Reconstruction metric using Entity and Event F1 scores. This specifically evaluates whether the model correctly identifies the "who" and "what" of the video.

Key Results and Insights

The KaVD model significantly outperformed both video-only and article-only baselines. On the manual Entity F1 metric, KaVD scored 22.1, compared to just 2.5 for standard video-only models. It also achieved statistically significant improvements in METEOR (10.2) and ROUGE-L (18.9) scores.

Qualitative analysis revealed that the video encoder and knowledge components are complementary: the video provides the "action" (verbs), while the retrieved articles provide the "actors" (nouns). For example, while a standard model might describe a generic bombing, the KaVD model can identify the specific group responsible by extracting details from the retrieved text.

Limitations and Future Work

The framework's performance is heavily dependent on the quality of retrieved articles and the accuracy of upstream IE tools. Irrelevant search results can lead to "hallucinated" entities in the captions. Future work aims to scale the dataset and explore more sophisticated knowledge-centric metrics to further refine the grounding of video descriptions in real-world context.

Summary updated

BibTeX
@inproceedings{whitehead2018incorporating,
  author = {Spencer Whitehead and Heng Ji and Mohit Bansal and Shih-Fu Chang and Clare R. Voss},
  title = {{Incorporating Background Knowledge into Video Description Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  url = {https://openreview.net/forum?id=ggeQlVNTIE0},
}
Figure