FIBER: Fill-in-the-Blanks as a Challenging Video Understanding Evaluation Framework

Santiago Castro, Ruoyao Wang, Pingxuan Huang, Ian Stewart, Oana Ignat, Nan Liu, Jonathan C. Stroud, Rada Mihalcea

TL;DR

This paper introduces FIBER, a novel "fill-in-the-blanks" evaluation framework for video understanding. By masking noun phrases in video captions and requiring models to predict them using both video and text, FIBER offers a more challenging and robust evaluation than existing methods like multiple-choice QA or open-ended captioning. The accompanying dataset of 28,000 videos and their annotated captions aims to push the boundaries of multimodal AI.

What Problem Does This Paper Address?

Current benchmarks for language-informed video understanding often fall short. Multiple-choice Video QA, for instance, is too easily gamed by models exploiting dataset biases rather than truly understanding the video. On the other hand, open-ended video captioning relies on subjective evaluation metrics that struggle with the inherent variability of human language, making automatic evaluation difficult. This leaves a gap for a more challenging and reliable evaluation method that truly tests a model's ability to integrate visual and textual information to understand video content.

Methods

The core of the paper is the FIBER framework, which frames video understanding as a fill-in-the-blanks task. Given a video and a caption with a masked noun phrase, the model must predict the missing phrase. To tackle this, the authors propose a multimodal encoder-decoder Transformer-based model. This model can be implemented with either an "early-fusion" (combining modalities early) or "late-fusion" (processing modalities separately before combining) architecture. The Transformer architecture is chosen for its effectiveness in encoding relationships within sequences, crucial for integrating visual and textual information.

Datasets

The primary dataset introduced is FIBER, comprising 28,000 videos. This dataset is built upon the existing VaTeX dataset, which provides 10-second YouTube video clips with 10 English and 10 Chinese captions each. For FIBER, noun phrases are systematically masked from the English captions of a subset of VaTeX, and then crowd workers provide multiple valid answers for these blanks. This multi-answer approach is key to addressing the limitations of single-ground-truth evaluations.

Key Insights and Ideas

  • Fill-in-the-blanks as a superior evaluation: The authors argue that masking noun phrases is a more challenging and informative task than multiple-choice QA or open-ended captioning, as it requires deeper reasoning and is less susceptible to superficial biases.
  • The importance of noun phrases: Noun phrases are highlighted as crucial for understanding visual content, as they often refer to people, objects, scenes, and events, requiring models to identify and reason about these elements in the video.
  • Diversity of answers is key: The FIBER dataset's strength lies in collecting multiple valid answers for each blank, reflecting the natural diversity of language and providing a more robust evaluation than relying on a single ground truth.
  • Multimodal models significantly outperform text-only: The experimental results clearly demonstrate that integrating visual information is essential for achieving good performance on this task, with multimodal models significantly outperforming text-only baselines.

Key Results

The proposed multimodal model (T5 + I3D) achieved a token-level F1 score of 71.4% on the FIBER validation set. This is significantly higher than the text-only baseline (T5 fine-tuned: 70.9%), indicating the importance of visual information. However, this score is still considerably lower than human performance (crowd workers achieved 82.5% F1), highlighting the challenging nature of the FIBER task. The paper also shows that the early-fusion multimodal model slightly outperforms the late-fusion model, and that using the whole video is more beneficial than relying on a single frame.

Limitations and Future Work

The authors acknowledge several limitations. Firstly, the task, while focusing on noun phrases, can indirectly capture other aspects like actions and object qualities, which might not always be the case. Secondly, the dataset primarily focuses on human-related activities, lacking broader representation of animals, nature, or technology. Thirdly, video availability on YouTube can be an issue over time. Finally, the annotation process, while designed to encourage diversity, might have led to some annotator bias due to incentives. Future work could involve expanding the dataset to cover more diverse topics, pre-training models on larger multimodal datasets, and exploring methods to mitigate gender bias in language models.

Summary updated

Abstract

We propose fill-in-the-blanks as a video understanding evaluation framework and introduce FIBER – a novel dataset consisting of 28,000 videos and descriptions in support of this evaluation framework. The fill-in-the-blanks setting tests a model’s understanding of a video by requiring it to predict a masked noun phrase in the caption of the video, given the video and the surrounding text. The FIBER benchmark does not share the weaknesses of the current state-of-the-art language-informed video understanding tasks, namely: (1) video question answering using multiple-choice questions, where models perform relatively well because they exploit linguistic biases in the task formulation, thus making our framework challenging for the current state-of-the-art systems to solve; and (2) video captioning, which relies on an open-ended evaluation framework that is often inaccurate because system answers may be perceived as incorrect if they differ in form from the ground truth. The FIBER dataset and our code are available at https://lit.eecs.umich.edu/fiber/.
BibTeX
@inproceedings{castro2022fiber,
  author = {Santiago Castro and Ruoyao Wang and Pingxuan Huang and Ian Stewart and Oana Ignat and Nan Liu and Jonathan C. Stroud and Rada Mihalcea},
  title = {{FIBER: Fill-in-the-Blanks as a Challenging Video Understanding Evaluation Framework}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.209},
}
Figure