Abstract: Jie Lei, Licheng Yu, Tamara Berg, Mohit Bansal. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
2020.
Predicting the next logical event in a dynamic scene is a hallmark of human intelligence, requiring a deep understanding of social dynamics, physical causality, and human intent. While AI has made significant progress in video captioning and question answering, future event prediction remains a formidable challenge. This paper introduces the Video-and-Language Future Event Prediction (VLEP) task and dataset. By employing an adversarial data collection process and integrating external commonsense knowledge, the authors reveal a substantial "intelligence gap" between state-of-the-art models and human performance.
Introduction
Traditional video-and-language tasks often focus on describing current actions or answering questions about the past. However, predicting the future requires "multimodal commonsense." For instance, seeing a detective reach for an evidence bag containing a phone with a text message implies they are about to read that text, not hand the evidence to a suspect.
Existing datasets for such tasks often suffer from "annotation artifacts"—unintentional linguistic biases that allow models to guess correctly using simple heuristics (like the presence of negation words) rather than true understanding. To address this, the authors propose VLEP, a benchmark designed to be resistant to such shortcuts through adversarial collection and matching.
The VLEP Dataset and Task
The VLEP dataset consists of 28,726 future event prediction examples derived from 10,234 clips. These clips are sourced from diverse environments, including popular TV shows like Friends and Grey’s Anatomy, as well as YouTube lifestyle vlogs.
In the VLEP task, a model is presented with a short premise video (averaging 6.1 seconds) and its associated dialogue. It must then choose which of two potential future events is more likely to occur. To aid in research and interpretability, the dataset also includes human-written rationales explaining why one event is more probable than the other.
Methodology
Adversarial Data Collection Pipeline
To ensure the dataset's difficulty and minimize linguistic bias, the researchers implemented a sophisticated "human-and-model-in-the-loop" collection strategy.
As shown in the pipeline, human writers were prompted to create a "more-likely" and a "less-likely" future event. If a baseline model could easily distinguish between the two, the writer was asked to revise the "less-likely" event to be more plausible and nuanced. This iterative process forced the creation of high-quality distractors that require actual video understanding to debunk. Additionally, "Adversarial Matching" was used to swap likely events from other clips into the current example as distractors, provided they were contextually relevant but incorrect.
Model Architecture and Commonsense Integration
The baseline model utilizes a multimodal transformer-based architecture. Video features are encoded using ResNet-152 (appearance) and ResNeXt-101 (motion), while text is processed via a RoBERTa-base encoder.
A key innovation is the pre-fine-tuning of the text encoder on ATOMIC, a large-scale knowledge graph of social "if-then" relations. This allows the model to leverage external world knowledge—such as understanding that if a person meets someone at a station, they might want to offer them a ride home—before attempting the specific VLEP task.
Key Results and Insights
The "Video Gap"
One of the most significant findings is the importance of visual information. Human performance reaches 76.25% when given only the dialogue, but climbs to 90.50% when the video is included. This confirms that the visual stream provides critical context for future prediction that text alone cannot capture.
Performance and the Intelligence Gap
The best-performing model (incorporating Video, Dialogue, and ATOMIC knowledge) achieved 67.46% accuracy. While this outperforms text-only baselines, it leaves a massive 23.04% gap compared to human performance.
The Role of Commonsense
Ablation studies confirmed that injecting knowledge from ATOMIC improved accuracy (from 66.96% to 67.46%). While the numerical gain appears modest, qualitative analysis suggests that such knowledge is vital for navigating complex social roles and intentions where the model otherwise fails.
Conclusion and Future Work
The VLEP benchmark highlights the current limitations of AI in multimodal reasoning. Models still struggle with complex social dynamics and the "why" behind human actions. Future research may move toward open-ended generation of future events or models capable of generating the rationales provided in the VLEP dataset, moving closer to the nuanced predictive capabilities of human intuition.
Summary updated