What is More Likely to Happen Next? Video-and-Language Future Event Prediction

Jie Lei, Licheng Yu, Tamara L. Berg, Mohit Bansal

Abstract: Jie Lei, Licheng Yu, Tamara Berg, Mohit Bansal. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

2020.

Predicting the next logical event in a dynamic scene is a hallmark of human intelligence, requiring a deep understanding of social dynamics, physical causality, and human intent. While AI has made significant progress in video captioning and question answering, future event prediction remains a formidable challenge. This paper introduces the Video-and-Language Future Event Prediction (VLEP) task and dataset. By employing an adversarial data collection process and integrating external commonsense knowledge, the authors reveal a substantial "intelligence gap" between state-of-the-art models and human performance.

Introduction

Traditional video-and-language tasks often focus on describing current actions or answering questions about the past. However, predicting the future requires "multimodal commonsense." For instance, seeing a detective reach for an evidence bag containing a phone with a text message implies they are about to read that text, not hand the evidence to a suspect.

Existing datasets for such tasks often suffer from "annotation artifacts"—unintentional linguistic biases that allow models to guess correctly using simple heuristics (like the presence of negation words) rather than true understanding. To address this, the authors propose VLEP, a benchmark designed to be resistant to such shortcuts through adversarial collection and matching.

The VLEP Dataset and Task

The VLEP dataset consists of 28,726 future event prediction examples derived from 10,234 clips. These clips are sourced from diverse environments, including popular TV shows like Friends and Grey’s Anatomy, as well as YouTube lifestyle vlogs.

Figure 2: Examples from the VLEP dataset showing premise videos, dialogues, and the choice between two future events with human rationales.

In the VLEP task, a model is presented with a short premise video (averaging 6.1 seconds) and its associated dialogue. It must then choose which of two potential future events is more likely to occur. To aid in research and interpretability, the dataset also includes human-written rationales explaining why one event is more probable than the other.

Methodology

Adversarial Data Collection Pipeline

To ensure the dataset's difficulty and minimize linguistic bias, the researchers implemented a sophisticated "human-and-model-in-the-loop" collection strategy.

Figure 4: The adversarial data collection pipeline where human writers receive model feedback to create more challenging distractor events.

As shown in the pipeline, human writers were prompted to create a "more-likely" and a "less-likely" future event. If a baseline model could easily distinguish between the two, the writer was asked to revise the "less-likely" event to be more plausible and nuanced. This iterative process forced the creation of high-quality distractors that require actual video understanding to debunk. Additionally, "Adversarial Matching" was used to swap likely events from other clips into the current example as distractors, provided they were contextually relevant but incorrect.

Model Architecture and Commonsense Integration

The baseline model utilizes a multimodal transformer-based architecture. Video features are encoded using ResNet-152 (appearance) and ResNeXt-101 (motion), while text is processed via a RoBERTa-base encoder.

A key innovation is the pre-fine-tuning of the text encoder on ATOMIC, a large-scale knowledge graph of social "if-then" relations. This allows the model to leverage external world knowledge—such as understanding that if a person meets someone at a station, they might want to offer them a ride home—before attempting the specific VLEP task.

Key Results and Insights

The "Video Gap"

One of the most significant findings is the importance of visual information. Human performance reaches 76.25% when given only the dialogue, but climbs to 90.50% when the video is included. This confirms that the visual stream provides critical context for future prediction that text alone cannot capture.

Performance and the Intelligence Gap

The best-performing model (incorporating Video, Dialogue, and ATOMIC knowledge) achieved 67.46% accuracy. While this outperforms text-only baselines, it leaves a massive 23.04% gap compared to human performance.

The Role of Commonsense

Ablation studies confirmed that injecting knowledge from ATOMIC improved accuracy (from 66.96% to 67.46%). While the numerical gain appears modest, qualitative analysis suggests that such knowledge is vital for navigating complex social roles and intentions where the model otherwise fails.

Conclusion and Future Work

The VLEP benchmark highlights the current limitations of AI in multimodal reasoning. Models still struggle with complex social dynamics and the "why" behind human actions. Future research may move toward open-ended generation of future events or models capable of generating the rationales provided in the VLEP dataset, moving closer to the nuanced predictive capabilities of human intuition.

Summary updated

BibTeX
@inproceedings{lei2020what,
  author = {Jie Lei and Licheng Yu and Tamara L. Berg and Mohit Bansal},
  title = {{What is More Likely to Happen Next? Video-and-Language Future Event Prediction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=CI3IGqKR1YU},
}
Figure