Abstract: Whitehead et al. (2018) present a novel framework for generating knowledge-rich video descriptions by leveraging external background knowledge. While standard models excel at describing actions, they often fail to identify specific entities and context. The proposed Knowledge-aware Video Description (KaVD) network addresses this by retrieving topically related news articles to ground descriptions in real-world facts.
Introduction: The Contextual Gap in Video Captioning
Standard video captioning models are remarkably proficient at describing general actions but often fail to identify specific actors and the broader context of an event. For instance, a model might describe a political broadcast as "a man is talking," missing the critical "who, where, and why" that define the video's significance.
The authors identify two primary challenges:
- Visual Insufficiency: Raw pixels often lack the information needed to identify specific named entities or the underlying causes of complex events.
- Contextual Gap: Recognizing a "protest" is insufficient if the model cannot distinguish between a climate rally in London and an election-related demonstration in Kenya.
The Knowledge-aware Video Description (KaVD) Framework
To bridge this gap, the authors propose the KaVD framework, which integrates external information into the generation process through three main stages:
1. Knowledge Retrieval & Extraction
Using video metadata—such as tags and upload dates—the system retrieves topically related news articles. Information Extraction (IE) tools then identify named entities (e.g., "Carles Puigdemont") and event types (e.g., "Demonstration") within these texts.
2. The Knowledge Gate
A novel gating mechanism in the decoder uses a one-hot vector representing the extracted entity and event types to "bias" the model. For example, detecting an "Attack" event encourages the decoder to prioritize terms like "bombing" or "explosion" over generic descriptors.
3. Entity Pointer Network
Because specific names appear infrequently in training data, the model employs a "soft switch" pointer network. This allows the system to choose between generating a common word from a fixed vocabulary or "pointing" to and copying a specific entity directly from the retrieved news documents.
Datasets and Evaluation
The researchers introduced a new News Video Dataset consisting of 2,883 videos from AFP’s YouTube channel. These descriptions are "knowledge-rich," averaging 2.1 named entities per sentence—a stark contrast to existing benchmarks like MSVD or MSR-VTT, which contain almost zero named entities.
To measure performance beyond simple fluency, the authors proposed a Knowledge Reconstruction metric using Entity and Event F1 scores. This specifically evaluates whether the model correctly identifies the "who" and "what" of the video.
Key Results and Insights
The KaVD model significantly outperformed both video-only and article-only baselines. On the manual Entity F1 metric, KaVD scored 22.1, compared to just 2.5 for standard video-only models. It also achieved statistically significant improvements in METEOR (10.2) and ROUGE-L (18.9) scores.
Qualitative analysis revealed that the video encoder and knowledge components are complementary: the video provides the "action" (verbs), while the retrieved articles provide the "actors" (nouns). For example, while a standard model might describe a generic bombing, the KaVD model can identify the specific group responsible by extracting details from the retrieved text.
Limitations and Future Work
The framework's performance is heavily dependent on the quality of retrieved articles and the accuracy of upstream IE tools. Irrelevant search results can lead to "hallucinated" entities in the captions. Future work aims to scale the dataset and explore more sophisticated knowledge-centric metrics to further refine the grounding of video descriptions in real-world context.
Summary updated