TL;DR
FIBER is a new video understanding benchmark that tasks models with filling in masked noun phrases in video captions. By providing a large-scale dataset of 28,000 videos with multiple human-annotated correct answers, it offers a more robust and challenging evaluation than traditional multiple-choice or open-ended captioning tasks.
What Problem Does This Paper Address?
Evaluating how well a machine "understands" a video is notoriously difficult. Current methods generally fall into two camps, both with significant flaws:
- Multiple-Choice Video QA: These are often "too easy" because models can exploit linguistic biases in the questions and answers to guess correctly without even looking at the video.
- Video Captioning: This is "too hard" to evaluate automatically. Because there are infinite ways to describe a scene, standard metrics (like BLEU or METEOR) often penalize perfectly valid descriptions that just happen to use different words than the ground truth.
The authors propose FIBER (Fill-in-the-Blanks as a Challenging Video Understanding Evaluation Framework) as a "Goldilocks" solution. It requires specific visual reasoning to fill a blank, but by collecting a wide array of human-validated synonyms and descriptions for each blank, it allows for fair, automated evaluation.
Methods
The researchers developed a multimodal encoder-decoder approach based on the T5 (Text-to-Text Transfer Transformer) architecture.
- Visual Features: They used pre-extracted I3D features, which capture both spatial and temporal information from video frames.
- Fusion Strategies: They tested two ways to combine video and text:
- Early-fusion: Concatenating tokenized text and video features into a single Transformer encoder.
- Late-fusion (Bate-fusion): Encoding text and video separately before fusing them in a joint encoder.
- The Task: The model receives a blanked caption (e.g., "A boy is trying to comb his hair while _____ dries it") and the corresponding video. It must then generate the missing noun phrase.
Datasets
- FIBER (New): The core contribution, consisting of 28,000 videos derived from the VaTeX dataset.
- VaTeX: The source for the original videos and captions.
- Human Annotation: Unlike previous "fill-in-the-blank" datasets that only have one correct answer, the authors used Amazon Mechanical Turk to collect an average of 13 correct answers per caption for the test and validation sets. This captures the natural diversity of language (e.g., "child," "kid," "young boy").
Key Insights and Ideas
- Why Noun Phrases? The authors specifically chose to mask noun phrases because they are harder to guess from surrounding text alone compared to verbs or adjectives. Predicting a noun often requires identifying specific objects, their properties (color, size), and their interactions in the video.
- Linguistic Diversity is Key: By clustering the top 100 answers using t-SNE, the authors showed that the dataset covers a wide range of semantic categories, from "Person-related" to "Objects," "Places," and "Body parts."
- The Human-Machine Gap: There remains a significant performance gap between the best models and human annotators, suggesting that FIBER is a challenging benchmark that will remain relevant as SOTA models improve.
- Visual Necessity: Multimodal models consistently outperformed text-only baselines, proving that the dataset successfully forces models to rely on visual information rather than just linguistic patterns.
Key Results
- Human Performance: Reached a token-level F1 score of 82.5% (using a leave-one-worker-out comparison).
- Best Model (T5 + I3D Early-fusion): Achieved an F1 score of 71.4% on the test set.
- Text-only Baseline: The best text-only model reached 70.9% F1. While the gap is small, the multimodal model's lead indicates it is successfully utilizing video data.
- Semantic Strengths: Models performed best on "Prepositions" and "Pronouns" (often guessable from text) but struggled significantly with "Actions" and "Passive entities," which require deeper visual reasoning.
Limitations and Future Work
- Gender Bias: The authors noted a slight skew toward male-related noun phrases (10.9% male vs 6.2% female in original labels), likely inherited from the source VaTeX dataset.
- Domain Specificity: The current dataset focuses heavily on human actions and sports. Future iterations could expand to more diverse domains like nature or technology.
- Higher-Order Reasoning: Models still struggle with concepts requiring complex reasoning, such as identifying an object based on an audio description ("the procedure described"). The authors suggest that future work should focus on these harder semantic categories.
Summary updated