FIBER: Fill-in-the-Blanks as a Challenging Video Understanding Evaluation Framework

Santiago Castro, Ruoyao Wang, Pingxuan Huang, Ian Stewart, Oana Ignat, Nan Liu, Jonathan C. Stroud, Rada Mihalcea

TL;DR

FIBER is a new video understanding benchmark that tasks models with filling in masked noun phrases in video captions. By providing a large-scale dataset of 28,000 videos with multiple human-annotated correct answers, it offers a more robust and challenging evaluation than traditional multiple-choice or open-ended captioning tasks.

What Problem Does This Paper Address?

Evaluating how well a machine "understands" a video is notoriously difficult. Current methods generally fall into two camps, both with significant flaws:

  1. Multiple-Choice Video QA: These are often "too easy" because models can exploit linguistic biases in the questions and answers to guess correctly without even looking at the video.
  2. Video Captioning: This is "too hard" to evaluate automatically. Because there are infinite ways to describe a scene, standard metrics (like BLEU or METEOR) often penalize perfectly valid descriptions that just happen to use different words than the ground truth.

The authors propose FIBER (Fill-in-the-Blanks as a Challenging Video Understanding Evaluation Framework) as a "Goldilocks" solution. It requires specific visual reasoning to fill a blank, but by collecting a wide array of human-validated synonyms and descriptions for each blank, it allows for fair, automated evaluation.

Methods

The researchers developed a multimodal encoder-decoder approach based on the T5 (Text-to-Text Transfer Transformer) architecture.

  • Visual Features: They used pre-extracted I3D features, which capture both spatial and temporal information from video frames.
  • Fusion Strategies: They tested two ways to combine video and text:
  • Early-fusion: Concatenating tokenized text and video features into a single Transformer encoder.
  • Late-fusion (Bate-fusion): Encoding text and video separately before fusing them in a joint encoder.
  • The Task: The model receives a blanked caption (e.g., "A boy is trying to comb his hair while _____ dries it") and the corresponding video. It must then generate the missing noun phrase.
The proposed multimodal architectures for the fill-in-the-blanks task, showing both early and late fusion approaches.

Datasets

  • FIBER (New): The core contribution, consisting of 28,000 videos derived from the VaTeX dataset.
  • VaTeX: The source for the original videos and captions.
  • Human Annotation: Unlike previous "fill-in-the-blank" datasets that only have one correct answer, the authors used Amazon Mechanical Turk to collect an average of 13 correct answers per caption for the test and validation sets. This captures the natural diversity of language (e.g., "child," "kid," "young boy").
The annotation interface used to collect diverse human answers for the FIBER dataset.

Key Insights and Ideas

  • Why Noun Phrases? The authors specifically chose to mask noun phrases because they are harder to guess from surrounding text alone compared to verbs or adjectives. Predicting a noun often requires identifying specific objects, their properties (color, size), and their interactions in the video.
  • Linguistic Diversity is Key: By clustering the top 100 answers using t-SNE, the authors showed that the dataset covers a wide range of semantic categories, from "Person-related" to "Objects," "Places," and "Body parts."
  • The Human-Machine Gap: There remains a significant performance gap between the best models and human annotators, suggesting that FIBER is a challenging benchmark that will remain relevant as SOTA models improve.
  • Visual Necessity: Multimodal models consistently outperformed text-only baselines, proving that the dataset successfully forces models to rely on visual information rather than just linguistic patterns.
A t-SNE visualization showing the semantic diversity of the answers provided by human annotators.

Key Results

  • Human Performance: Reached a token-level F1 score of 82.5% (using a leave-one-worker-out comparison).
  • Best Model (T5 + I3D Early-fusion): Achieved an F1 score of 71.4% on the test set.
  • Text-only Baseline: The best text-only model reached 70.9% F1. While the gap is small, the multimodal model's lead indicates it is successfully utilizing video data.
  • Semantic Strengths: Models performed best on "Prepositions" and "Pronouns" (often guessable from text) but struggled significantly with "Actions" and "Passive entities," which require deeper visual reasoning.

Limitations and Future Work

  • Gender Bias: The authors noted a slight skew toward male-related noun phrases (10.9% male vs 6.2% female in original labels), likely inherited from the source VaTeX dataset.
  • Domain Specificity: The current dataset focuses heavily on human actions and sports. Future iterations could expand to more diverse domains like nature or technology.
  • Higher-Order Reasoning: Models still struggle with concepts requiring complex reasoning, such as identifying an object based on an audio description ("the procedure described"). The authors suggest that future work should focus on these harder semantic categories.

Summary updated

Abstract

We propose fill-in-the-blanks as a video understanding evaluation framework and introduce FIBER -- a novel dataset consisting of 28,000 videos and descriptions in support of this evaluation framework. The fill-in-the-blanks setting tests a model's understanding of a video by requiring it to predict a masked noun phrase in the caption of the video, given the video and the surrounding text. The FIBER benchmark does not share the weaknesses of the current state-of-the-art language-informed video understanding tasks, namely: (1) video question answering using multiple-choice questions, where models perform relatively well because they exploit linguistic biases in the task formulation, thus making our framework challenging for the current state-of-the-art systems to solve; and (2) video captioning, which relies on an open-ended evaluation framework that is often inaccurate because system answers may be perceived as incorrect if they differ in form from the ground truth. The FIBER dataset and our code are available at https://lit.eecs.umich.edu/fiber/.
BibTeX
@inproceedings{castro2022fiber,
  author = {Santiago Castro and Ruoyao Wang and Pingxuan Huang and Ian Stewart and Oana Ignat and Nan Liu and Jonathan C. Stroud and Rada Mihalcea},
  title = {{FIBER: Fill-in-the-Blanks as a Challenging Video Understanding Evaluation Framework}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.209},
}
Figure