MoviePuzzle: Visual Narrative Reasoning through Multimodal Order Learning

Jianghui Wang, Yuxuan Wang, Dongyan Zhao, Zilong Zheng

TL;DR

Researchers have introduced MoviePuzzle, a benchmark challenging AI to reorder scrambled movie clips. To solve it, they developed Hierarchical Contrastive Movie Clustering (HCMC), a model that leverages the natural hierarchy of film (frames, shots, and scenes) to outperform state-of-the-art video-language models in understanding long-form narrative flow.

The Challenge of Visual Narrative Reasoning

While AI excels at identifying simple actions in short clips, it struggles with the "big picture." Most existing models treat video as a flat sequence of frames, overlooking the narrative structure—the logical and causal links that turn images into a story. In long-form content like movies, meaning is embedded in dialogue, transitions between camera angles (shots), and shifts in location (scenes). The authors argue that "Visual Narrative Reasoning" (VNR) is a critical missing piece for achieving true holistic video understanding.

The MoviePuzzle Benchmark

The core task of MoviePuzzle is temporal reordering. Given a set of shuffled vision-utterance pairs (a frame and its corresponding subtitle), a model must predict the original temporal index for each.

The MoviePuzzle task requires models to identify temporal, causal, and dialogue-based relationships between shuffled segments.

The dataset includes 10,031 movie clips from 228 films, featuring 135,837 frames paired with English subtitles. It is meticulously annotated with 15,000 scene boundaries and 81,000 shot boundaries, providing a robust foundation for training models to understand cinematic structure.

Hierarchical Contrastive Movie Clustering (HCMC)

The HCMC model is designed to mimic human narrative comprehension by operating across three layers:

  1. Frame-Level: Extracting visual and textual features for individual moments using CLIP-based encoders.
  2. Shot-Level: Aggregating frames that share a fixed camera angle.
  3. Scene-Level: Grouping shots that share an environmental context or event.
HCMC models the natural hierarchy of film, organizing frames into shots and shots into scenes to capture narrative flow.

HCMC uses pairwise learning to decide the relative order of segments and contrastive learning to help the model distinguish boundaries. During testing, it employs a top-down/bottom-up inference strategy, clustering frames into scenes and shots before using a beam search to find the most probable chronological path.

Key Insights and Results

  • Dialogue as a Narrative Anchor: Pure language models performed remarkably well, suggesting that the script's flow is often a stronger temporal indicator than visual cues alone.
  • Vision Models are "Temporally Blind": Standard vision-only models (like VideoMAE) performed poorly, highlighting a deficiency in how current models process temporal logic in images.
  • Hierarchy is Essential: Treating a movie as a flat list of frames is computationally expensive and logically flawed. By explicitly modeling shots and scenes, HCMC captures the narrative layers humans intuitively use.
  • Benchmark Performance: HCMC achieved an Ordering Score (OS) of 55.40, significantly outperforming random baselines and large multimodal models like VideoLLaVA and GPT-4 in zero-shot settings.
  • Downstream Gains: Training a model to "solve the puzzle" improved its performance on other tasks, such as matching movie segments to written synopses.

Limitations and Future Work

The authors identify several areas for growth, including maintaining global consistency across very long sequences and handling artistic editing like montages or flashbacks that intentionally break visual continuity. Future work may involve integrating these reasoning capabilities into generative video models to ensure AI-generated stories are logically sound.

Summary updated

Abstract

We introduce MoviePuzzle, a novel challenge that targets visual narrative reasoning and holistic movie understanding. Despite the notable progress that has been witnessed in the realm of video understanding, most prior works fail to present tasks and models to address holistic video understanding and the innate visual narrative structures existing in long-form videos. To tackle this quandary, we put forth MoviePuzzle task that amplifies the temporal feature learning and structure learning of video models by reshuffling the shot, frame, and clip layers of movie segments in the presence of video-dialogue information. We start by establishing a carefully refined dataset based on MovieNet by dissecting movies into hierarchical layers and randomly permuting the orders. Besides benchmarking the MoviePuzzle with prior arts on movie understanding, we devise a Hierarchical Contrastive Movie Clustering (HCMC) model that considers the underlying structure and visual semantic orders for movie reordering. Specifically, through a pairwise and contrastive learning approach, we train models to predict the correct order of each layer. This equips them with the knack for deciphering the visual narrative structure of movies and handling the disorder lurking in video data. Experiments show that our approach outperforms existing state-of-the-art methods on the \MoviePuzzle benchmark, underscoring its efficacy.
BibTeX
@misc{wang2023moviepuzzle,
  author = {Jianghui Wang and Yuxuan Wang and Dongyan Zhao and Zilong Zheng},
  title = {{MoviePuzzle: Visual Narrative Reasoning through Multimodal Order Learning}},
  year = {2023},
  eprint = {2306.02252},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.02252},
}
Figure