Abstract
Automated movie trailer generation is a challenging task that requires selecting high-impact shots and arranging them to create a compelling narrative synchronized with music. We present MMSC (Multi-Modal Semantic Consistency), a weakly-supervised framework that learns directly from movie-trailer pairs without expensive manual annotations. By aligning visual shots with music energy and movie metadata through a novel "pseudo-score" system, MMSC selects and reorders key moments to produce trailers that are both semantically coherent and emotionally resonant.
Introduction
Creating a movie trailer is a sophisticated editorial feat. Unlike simple video summarization, which typically maintains chronological order, trailers require selecting "high-impact" shots and rearranging them to build suspense, often synchronized to a specific musical rhythm. Existing automated methods face three major hurdles: data scarcity (the high cost of frame-level labels), the modality gap (ignoring the interplay between music, dialogue, and plot), and poor generalization across diverse movie genres.
MMSC addresses these challenges by treating trailer generation as a multi-modal alignment problem, leveraging existing official trailers as a source of weak supervision to bridge the gap between raw movie footage and professional-grade edits.
Methodology
The MMSC framework consists of three primary stages: multi-modal encoding, weakly-supervised learning via pseudo-scores, and post-processing for narration.
1. Multi-Modal Encoding
The model utilizes pre-trained foundation models to extract features across different data types. ImageBind is employed for visual and textual (metadata) features, while CLAP handles acoustic features. Cross-attention mechanisms then fuse these modalities, allowing the visual representation of a shot to be informed by the movie's genre tags and plot keywords.
2. Weakly-Supervised Learning via Pseudo-Scores
To avoid the need for manual labels, MMSC generates "pseudo-scores" from official trailers to guide training:
- Trailerness Pseudo-Score: Measures the similarity of a movie shot to shots in the official trailer, teaching the model what to select.
- Emotion Pseudo-Score: Derived from the energy (MFCCs) of the trailer's music, teaching the model how to sort shots by matching high-intensity visuals with high-energy audio.
3. Post-Processing and Narration
To achieve a professional feel, the system integrates Large Language Models (LLMs). DeepSeek-V3 analyzes subtitles to extract narrative sentences, while MiniCPM-V 2.6 generates captions for selected shots. Dynamic programming then aligns this narration with visual shots based on semantic similarity.
Experimental Setup
The researchers curated the MMSC Dataset, comprising 500 movies and 922 trailers spanning 18 genres and 30 years. Evaluation was conducted on Test-74 (a subset of 74 pairs) and Test-8 (a standard benchmark used in prior literature like Muvee and IPOT) to ensure fair comparison.
Key Insights
- Music as a Temporal Guide: By forcing the model to predict an "emotion score" that matches the music's energy profile, the resulting trailer naturally follows the rhythmic "ebb and flow" of a professional edit.
- Metadata as a "North Star": Ablation studies confirmed that including plot keywords and genre labels significantly boosts performance, helping the model distinguish between similar-looking shots in different contexts (e.g., horror vs. noir).
- The Power of Weak Supervision: The results demonstrate that models can learn the "essence" of a trailer by treating official edits as noisy labels, bypassing the need for frame-by-frame human annotation.
Key Results
MMSC outperformed state-of-the-art methods across both objective and subjective metrics. On the Test-74 set, MMSC achieved an F1-score of 0.1991, significantly higher than the previous best (IPOT at 0.1357). It also demonstrated superior Pairwise Agreement Accuracy (AA), indicating better shot sequencing.
In a subjective user study with 30 volunteers, MMSC consistently scored closest to the "Official Trailer" (the gold standard) across categories such as Theme, Rhythm, and Appropriateness.
Limitations and Future Work
While MMSC represents a significant step forward, its performance can dip on "atypical" trailers that subvert standard editing tropes. Furthermore, while LLM-based narration improves flow, the model does not yet fully grasp complex three-act narrative structures. Future work will explore integrating custom music generation models to create scores tailored specifically to the selected visual highlights.
Summary updated