Select and Summarize: Scene Saliency for Movie Script Summarization

Rohit Saxena, Frank Keller

TL;DR

Summarizing long movie scripts is tough for AI due to their length and the need to capture narrative essence. This paper introduces a novel two-stage approach: first, it identifies "salient" scenes (those crucial to the plot) using a dedicated classification model, and then it generates a summary using only these key scenes. This method significantly outperforms existing approaches, demonstrating the power of targeted content selection for long-form summarization.

What Problem Does This Paper Address?

Current large language models (LLMs) struggle with summarizing very long documents like movie scripts. This is primarily due to memory constraints and their difficulty in attending to relevant information spread across extensive text. While abstractive summarization aims to generate concise summaries, applying it to movie scripts is particularly challenging because scripts are structured into numerous scenes, and only a subset of these scenes are truly important for understanding the overall narrative. Identifying these "salient" scenes automatically is a significant hurdle, as existing methods often fail to scale or lack appropriate evaluation datasets.

Methods

The paper proposes a two-stage pipeline:

  1. Scene Saliency Classification: This stage identifies which scenes in a movie script are salient.
  2. Dataset Creation: The authors first create a new dataset, MENSA, by having human annotators align Wikipedia plot summaries with movie script scenes. This provides "gold-standard" labels for scene saliency.
  3. Silver-Standard Generation: To scale up, they use an existing alignment method (Mirza et al., 2021) to generate "silver-standard" labels for a larger dataset.
  4. Model Architecture: A transformer-based sequence labeling model is trained to predict binary saliency labels for each scene. This model uses a scene encoder to represent individual scenes and a transformer block to capture contextual relationships between scenes.
  5. Baselines: They compare their model against Majority Class, unsupervised TextRank, and a supervised Bi-LSTM.
  6. Abstractive Summarization using Salient Scenes: Once salient scenes are identified, they are used to generate the summary.
  7. Model Architecture: A Longformer Encoder-Decoder (LED) model is employed, which is designed to handle long input sequences efficiently.
  8. Input: Instead of the entire movie script, only the identified salient scenes are fed into the summarization model.
  9. Baselines: They compare their approach against various methods, including lead-N, FLAN-T5, Vicuna, GPT-3.5, and other state-of-the-art summarization models.

Datasets

  • MENSA: A newly introduced dataset consisting of 100 movies with human-annotated salient scenes derived from Wikipedia plot summaries. This dataset is crucial for evaluating scene saliency identification.
  • ScriptBase: A larger corpus of movie scripts used for training the saliency classification model and for generating silver-standard labels. Movies from MENSA were excluded from the training set.
  • SummScreenFD: Used for evaluating the scene saliency classifier in a zero-shot setting, this dataset consists of TV show episodes and their recaps.

Key Insights and Ideas

  • Salience as a Proxy for Importance: The core idea is that scenes mentioned in a plot summary are likely the most important for understanding the narrative. This operationalization of "salience" is key to their approach.
  • Two-Stage Pipeline Effectiveness: The paper strongly suggests that a two-stage process—first identifying salient content and then summarizing—is more effective for long-form documents than end-to-end methods that struggle with scale.
  • Transformer Power for Context: The transformer architecture proves effective in capturing inter-scene relationships, which is vital for accurately classifying scene saliency in long scripts.
  • Content Selection Reduces Input Size: By focusing only on salient scenes, the input length for the summarization model is significantly reduced, leading to better performance and potentially lower computational costs.
  • QA-based Evaluation is Crucial: Traditional metrics like ROUGE and BERTScore are useful but don't fully capture factual consistency. The use of a Question-Answering evaluation (QAEval) provides a more robust assessment of summary quality.

Key Results

  • Scene Saliency Classification: The proposed transformer-based scene saliency model achieved state-of-the-art results, outperforming baselines with macro-averaged precision, recall, and F1 scores of 68.38, 68.13, and 68.01, respectively.
  • Summarization Performance: The "Select and Summarize" model, using salient scenes, achieved state-of-the-art ROUGE scores (e.g., 49.98 ROUGE-1 F1) on the ScriptBase corpus.
  • Comparison to Full Text: Summaries generated using only salient scenes outperformed those generated using the full movie script by a significant margin (e.g., 3.83 ROUGE-1 points improvement).
  • QAEval Results: The model achieved a mean F1 of 29.42 and a mean exact match of 20.05% on the QAEval metric, demonstrating superior factual consistency compared to full-text summarization.

Limitations and Future Work

  • Definition of Salience: The authors acknowledge that their definition of scene saliency (based on mention in summaries) is limited. Future work could explore other factors like character presence, visual impact, or narrative importance.
  • Script Quality: Some scripts in the dataset might not be final production versions, potentially impacting summary quality.
  • Pipeline Errors: The two-stage pipeline means that errors in saliency classification can propagate to the summarization stage.
  • Human Evaluation: Evaluating summaries of long texts is challenging and requires human evaluators to read extensive material. Future work should include more human evaluations.
  • Exploring Other Salience Factors: Investigating factors beyond summary mentions for scene saliency.
  • End-to-End Models: Developing end-to-end models that can handle long contexts more effectively.

Summary updated

Abstract

Abstractive summarization for long-form narrative texts such as movie scripts is challenging due to the computational and memory constraints of current language models. A movie script typically comprises a large number of scenes; however, only a fraction of these scenes are salient, i.e., important for understanding the overall narrative. The salience of a scene can be operationalized by considering it as salient if it is mentioned in the summary. Automatically identifying salient scenes is difficult due to the lack of suitable datasets. In this work, we introduce a scene saliency dataset that consists of human-annotated salient scenes for 100 movies. We propose a two-stage abstractive summarization approach which first identifies the salient scenes in script and then generates a summary using only those scenes. Using QA-based evaluation, we show that our model outperforms previous state-of-the-art summarization methods and reflects the information content of a movie more accurately than a model that takes the whole movie script as input.
BibTeX
@inproceedings{saxena2024select,
  author = {Rohit Saxena and Frank Keller},
  title = {{Select and Summarize: Scene Saliency for Movie Script Summarization}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.218},
}
Figure