TL;DR
This paper proposes a novel deep learning approach to predict movie success using only plot summaries, bypassing the need for post-production data like reviews or trailers. By leveraging ELMO embeddings and sentiment analysis, the models achieve promising results, suggesting that the narrative itself holds significant predictive power for a film's commercial and critical reception.
What Problem Does This Paper Address?
The film industry is a high-stakes business where predicting a movie's success early on is crucial for investment decisions. Traditional methods often rely on data available only after production, such as reviews, trailers, or social media buzz. This leaves a critical gap: how can investors gauge a movie's potential success before it's even made, based solely on its core creative element – the story? This paper tackles this challenge by exploring whether a movie's plot summary, a textual representation of its narrative, can be used to predict its success.
Methods
The authors propose a deep learning framework that processes movie plot summaries. The core of their approach involves:
- ELMO Embeddings: Instead of static word embeddings (like Word2vec or GloVe), they use ELMO (Embeddings from Language Models). ELMO generates contextualized word embeddings, meaning the representation of a word changes based on its surrounding words, capturing nuances and polysemy. This is crucial for understanding the rich language of plot summaries.
- Sentiment Score Extraction: They employ the NLTK's Vader sentiment analyzer to extract sentiment scores (positive, negative, neutral, and compound) for each sentence in the plot summary. This captures the emotional arc and tone of the narrative.
- Merged Deep Learning Models: Two main architectures are explored:
- Merged 1D CNN: This model combines ELMO embeddings with a 1D Convolutional Neural Network (CNN) and sentiment scores. The CNN is used to extract local features from the text, while the sentiment scores provide additional context.
- Merged Residual Bidirectional LSTM: This architecture integrates ELMO embeddings with a bidirectional Long Short-Term Memory (LSTM) network. LSTMs are well-suited for sequential data like text, and the bidirectional nature allows them to process information from both past and future sentences, capturing long-range dependencies.
- Classification: The outputs from these models are then fed into a classification layer to predict whether a movie will be successful or not.
Datasets
The primary dataset used is the CMU Movie Summary Corpus. This corpus contains over 42,000 movie plot summaries. For evaluation, the authors crawled Rotten Tomatoes scores to define movie success. They created three distinct evaluation datasets:
- Audience Score: Movies classified as "popular" (score > 75) or "not popular" (score < 60).
- Critics Score: Movies classified as "well-made" (score > 75) or "not well-made" (score < 60).
- Audience & Critics Score: Movies classified as successful if they met both "popular" and "well-made" criteria (scores > 75 for both).
Movies with scores between 60 and 75 were filtered out, as were movies lacking review scores.
Key Insights and Ideas
- Narrative as a Predictor: The most significant insight is that a movie's plot summary alone, without any external information like cast or budget, can be a surprisingly strong predictor of its success. This suggests that the inherent quality and appeal of the story are fundamental drivers of a film's reception.
- Contextual Embeddings Matter: The use of ELMO embeddings highlights the importance of understanding word meaning in context. This is likely more effective than static embeddings for capturing the nuances of storytelling.
- Sentiment Arc is Informative: The sentiment flow analysis (Figures 3 and 4) suggests that successful movies often exhibit more dynamic sentiment fluctuations, while unsuccessful ones tend to have flatter sentiment arcs. This indicates that the emotional journey of the plot might be a key factor.
- "Not Successful" is Easier to Predict: The models generally performed better at predicting "not popular" or "not well-made" movies than their successful counterparts. This could imply that certain narrative patterns or themes are more reliably indicative of failure than success.
Key Results
- The proposed deep learning models (CNN and LSTM merged with ELMO and sentiment scores) consistently outperform a majority class baseline.
- For audience scores, the best performance was achieved by the CNN model for "all genres" with an F1 score of 0.79. ELMO performed best in specific genres like drama and thriller.
- For critics scores, the CNN model achieved the highest F1 score of 0.72 for predicting "well-made" drama movies. The CNN also excelled in predicting "not well-made" movies in thriller and comedy genres (F1 of 0.75 and 0.77 respectively).
- When combining audience and critics scores, the ELMO model achieved the highest F1 scores of 0.68 for "successful" and 0.70 for "not successful" movies.
Limitations and Future Work
- Data Bias: The authors acknowledge that plot summaries obtained from sources like Wikipedia might be written by fans, potentially introducing bias. Some summaries might be written by people who "like or value the movie," complicating the prediction task.
- Limited Scope of Success: The current definition of success is binary (successful/not successful) based on specific score thresholds. The authors suggest extending the model to predict the magnitude of a movie's success using linear regression.
- Genre-Specific Performance: While the models show promise, performance varies across genres. Future work could focus on improving predictions for genres where current performance is lower.
- Future Directions: The authors propose further investigating the efficacy of their approach in specific genres (thriller, comedy) and extending the model to predict different levels of success.
Summary updated