Weakly-Supervised Movie Trailer Generation Driven by Multi-Modal Semantic Consistency

Sidan Zhu, Yutong Wang, Hongteng Xu, Dixin Luo

Abstract

Automated movie trailer generation is a challenging task that requires selecting high-impact shots and arranging them to create a compelling narrative synchronized with music. We present MMSC (Multi-Modal Semantic Consistency), a weakly-supervised framework that learns directly from movie-trailer pairs without expensive manual annotations. By aligning visual shots with music energy and movie metadata through a novel "pseudo-score" system, MMSC selects and reorders key moments to produce trailers that are both semantically coherent and emotionally resonant.

Introduction

Creating a movie trailer is a sophisticated editorial feat. Unlike simple video summarization, which typically maintains chronological order, trailers require selecting "high-impact" shots and rearranging them to build suspense, often synchronized to a specific musical rhythm. Existing automated methods face three major hurdles: data scarcity (the high cost of frame-level labels), the modality gap (ignoring the interplay between music, dialogue, and plot), and poor generalization across diverse movie genres.

MMSC addresses these challenges by treating trailer generation as a multi-modal alignment problem, leveraging existing official trailers as a source of weak supervision to bridge the gap between raw movie footage and professional-grade edits.

Methodology

The MMSC framework consists of three primary stages: multi-modal encoding, weakly-supervised learning via pseudo-scores, and post-processing for narration.

1. Multi-Modal Encoding

The model utilizes pre-trained foundation models to extract features across different data types. ImageBind is employed for visual and textual (metadata) features, while CLAP handles acoustic features. Cross-attention mechanisms then fuse these modalities, allowing the visual representation of a shot to be informed by the movie's genre tags and plot keywords.

2. Weakly-Supervised Learning via Pseudo-Scores

To avoid the need for manual labels, MMSC generates "pseudo-scores" from official trailers to guide training:

  • Trailerness Pseudo-Score: Measures the similarity of a movie shot to shots in the official trailer, teaching the model what to select.
  • Emotion Pseudo-Score: Derived from the energy (MFCCs) of the trailer's music, teaching the model how to sort shots by matching high-intensity visuals with high-energy audio.
Overview of the MMSC framework including training, pseudo-score generation, and inference stages.

3. Post-Processing and Narration

To achieve a professional feel, the system integrates Large Language Models (LLMs). DeepSeek-V3 analyzes subtitles to extract narrative sentences, while MiniCPM-V 2.6 generates captions for selected shots. Dynamic programming then aligns this narration with visual shots based on semantic similarity.

Experimental Setup

The researchers curated the MMSC Dataset, comprising 500 movies and 922 trailers spanning 18 genres and 30 years. Evaluation was conducted on Test-74 (a subset of 74 pairs) and Test-8 (a standard benchmark used in prior literature like Muvee and IPOT) to ensure fair comparison.

Key Insights

  • Music as a Temporal Guide: By forcing the model to predict an "emotion score" that matches the music's energy profile, the resulting trailer naturally follows the rhythmic "ebb and flow" of a professional edit.
  • Metadata as a "North Star": Ablation studies confirmed that including plot keywords and genre labels significantly boosts performance, helping the model distinguish between similar-looking shots in different contexts (e.g., horror vs. noir).
  • The Power of Weak Supervision: The results demonstrate that models can learn the "essence" of a trailer by treating official edits as noisy labels, bypassing the need for frame-by-frame human annotation.

Key Results

MMSC outperformed state-of-the-art methods across both objective and subjective metrics. On the Test-74 set, MMSC achieved an F1-score of 0.1991, significantly higher than the previous best (IPOT at 0.1357). It also demonstrated superior Pairwise Agreement Accuracy (AA), indicating better shot sequencing.

In a subjective user study with 30 volunteers, MMSC consistently scored closest to the "Official Trailer" (the gold standard) across categories such as Theme, Rhythm, and Appropriateness.

Subjective evaluation results showing MMSC (Ours) outperforming baselines and approaching the quality of Real Trailers (RT) in appropriateness.

Limitations and Future Work

While MMSC represents a significant step forward, its performance can dip on "atypical" trailers that subvert standard editing tropes. Furthermore, while LLM-based narration improves flow, the model does not yet fully grasp complex three-act narrative structures. Future work will explore integrating custom music generation models to create scores tailored specifically to the selected visual highlights.

Summary updated

Abstract

As an essential movie promotional tool, trailers are designed to capture the audience's interest through the skillful editing of key movie shots. Although some attempts have been made for automatic trailer generation, existing methods often rely on predefined rules or manual fine-grained annotations and fail to fully leverage the multi-modal information of movies, resulting in unsatisfactory trailer generation results. In this study, we introduce a weakly-supervised trailer generation method driven by multi-modal semantic consistency. Specifically, we design a multi-modal trailer generation framework that selects and sorts key movie shots based on input music and movie metadata (e.g., category tags and plot keywords) and adds narration to the generated trailer based on movie subtitles. We utilize two pseudo-scores derived from the proposed framework as labels and thus train the model under a weakly-supervised learning paradigm, ensuring trailerness consistency for key shot selection and emotion consistency for key shot sorting, respectively. As a result, we can learn the proposed model solely based on movie-trailer pairs without any fine-grained annotations. Both objective experimental results and subjective user studies demonstrate the superior performance of our method over previous works. The code is available at https://github.com/Dixin-Lab/MMSC.
BibTeX
@inproceedings{zhu2025weakly,
  author = {Sidan Zhu and Yutong Wang and Hongteng Xu and Dixin Luo},
  title = {{Weakly-Supervised Movie Trailer Generation Driven by Multi-Modal Semantic Consistency}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/1137},
}
Figure