What is More Likely to Happen Next? Video-and-Language Future Event Prediction

Jie Lei, Licheng Yu, Tamara L. Berg, Mohit Bansal

Introduction

Humans possess a remarkable ability to watch a scene and infer what happens next. This "future-seeing" capability isn't just about predicting pixels; it involves understanding social dynamics, dialogue context, and the "commonsense" rules of the world. Researchers from UNC Chapel Hill have introduced Video-and-Language Future Event Prediction (VLEP), a new task that challenges AI to perform this high-level reasoning.

The VLEP Dataset

To support this research, the authors released the VLEP dataset, which includes 28,726 future event prediction examples derived from 10,234 diverse video clips. These clips are sourced from popular TV shows (e.g., The Big Bang Theory, Castle) and YouTube lifestyle vlogs.

Examples of the VLEP task from TV shows and lifestyle vlogs.

Each example consists of a video premise with aligned dialogue and two potential future events. The model must determine which event is more likely to occur based on the provided context.

Adversarial Data Collection

To ensure the dataset is non-trivial, the authors used an adversarial human-and-model-in-the-loop procedure. Early datasets often contain "annotation artifacts"—biases in how humans write unlikely events (e.g., using specific negative words) that allow models to "cheat" by ignoring the video premise.

The adversarial human-and-model-in-the-loop data collection pipeline.

In this pipeline, crowd-workers were prompted to rewrite their "less-likely" events if a model could easily guess the correct answer without visual context. This process significantly increased the difficulty and quality of the benchmark, forcing models to rely on true multimodal understanding.

Methodology: Fusing Vision, Language, and Commonsense

The authors developed a transformer-based baseline model that integrates three streams of information:

  1. Video Encoding: Uses ResNet-152 and ResNeXt-101 to capture both spatial appearance and temporal motion.
  2. Text Encoding: Employs RoBERTa-base to process dialogue and candidate events.
  3. Commonsense Injection: The RoBERTa encoder is fine-tuned on the ATOMIC dataset, which contains over 400,000 social commonsense inferences. This gives the model "world knowledge" about human intentions and reactions.

These features are fused in a Multimodal Transformer Encoder, allowing the model to reason across modalities before making a final prediction via a Softmax layer.

Key Results and Insights

  • The Human-AI Gap: The best-performing model achieved 67.46% accuracy, while humans reached 90.50%. This 23% gap highlights the immense room for improvement in multimodal reasoning.
  • Multimodality is Mandatory: The study found that humans and models both perform significantly better when they have access to both video and dialogue. Visual context provides critical information that text alone cannot capture.
  • Commonsense as a Bridge: Injecting knowledge from ATOMIC provided a measurable performance boost, suggesting that high-level prediction requires more than just processing the current scene; it requires understanding general patterns of human behavior.

Conclusion

The VLEP task and dataset provide a rigorous benchmark for the next generation of AI models. While current architectures can leverage multimodal cues and commonsense knowledge to some extent, they still fall far short of human-level inference. Future research will need to focus on deeper video-dialogue alignment and more sophisticated reasoning over external knowledge bases.

Summary updated

Abstract

Given a video with aligned dialogue, people can often infer what is more likely to happen next. Making such predictions requires not only a deep understanding of the rich dynamics underlying the video and dialogue, but also a significant amount of commonsense knowledge. In this work, we explore whether AI models are able to learn to make such multimodal commonsense next-event predictions. To support research in this direction, we collect a new dataset, named Video-and-Language Event Prediction (VLEP), with 28,726 future event prediction examples (along with their rationales) from 10,234 diverse TV Show and YouTube Lifestyle Vlog video clips. In order to promote the collection of non-trivial challenging examples, we employ an adversarial human-and-model-in-the-loop data collection procedure. We also present a strong baseline incorporating information from video, dialogue, and commonsense knowledge. Experiments show that each type of information is useful for this challenging task, and that compared to the high human performance on VLEP, our model provides a good starting point but leaves large room for future work. Our dataset and code are available at: https://github.com/jayleicn/VideoLanguageFuturePred
BibTeX
@inproceedings{lei2020what,
  author = {Jie Lei and Licheng Yu and Tamara L. Berg and Mohit Bansal},
  title = {{What is More Likely to Happen Next? Video-and-Language Future Event Prediction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=CI3IGqKR1YU},
}
Figure