TL;DR
This paper introduces a novel dataset of films paired with their soundtracks and subtitles to empirically analyze the relationship between music and narrative in film. They find that musical attributes of soundtracks are predictive of film ratings, even after controlling for topical content, suggesting music plays a significant role in audience reception.
What Problem Does This Paper Address?
Film is a multimodal medium where narrative is conveyed through various channels, including visuals, dialogue, and music. While much research in computational linguistics and computer vision has focused on text and image modalities, the role of music in film storytelling has been largely overlooked. This paper aims to bridge this gap by computationally modeling the connection between a film's narrative content (derived from scripts) and its soundtrack's musical characteristics, and subsequently, how these musical elements influence audience reception as measured by user ratings.
Methods
The authors construct a large dataset by linking film metadata from IMDB with subtitle data from OpenSubtitles2016 and audio features from Spotify. They employ Latent Dirichlet Allocation (LDA) to identify 50 latent topics within film scripts, representing the narrative content. These topic distributions are then used to model the relationship with various audio features of the soundtrack, including mode (major/minor), tempo, danceability, instrumentalness, and acousticness. For analyzing the impact on user ratings, they utilize a causal inference technique called coarsened exact matching to control for confounding factors like topical content and genre.
Datasets
- OpenSubtitles2016: A dataset of parallel texts for film subtitles, providing scripts for a large number of movies and TV shows.
- IMDB: Used to extract metadata for films, including title, year, average user rating, and genre.
- Spotify API: Used to obtain structured audio information (features) for songs.
The paper introduces a new benchmark dataset by pairing 41,143 films with their soundtracks and subtitles.
Key Insights and Ideas
- Music as a Narrative Device: The research strongly suggests that soundtracks are not merely background noise but active contributors to film storytelling, influencing how audiences perceive and rate a movie.
- Topic-Music Correlation: Specific narrative themes or topics within a film's script are demonstrably associated with particular musical attributes in its soundtrack. For instance, "captain ship" topics correlate with major keys, while themes of separation or FBI agents are linked to minor keys.
- Musical Attributes Predict Ratings: Beyond just fitting the narrative, certain musical characteristics of a soundtrack can independently predict a film's reception. This implies there are broadly effective musical elements for storytelling.
- Danceability and Acousticness Impact: Highly danceable soundtracks tend to lower film ratings, while acoustic and instrumental soundtracks (with no vocals) are associated with higher ratings, even when controlling for other factors.
Key Results
- The dataset comprises 41,143 films, 189,340 songs (96,526 unique), and paired scripts.
- Topic-Music Relationships:
- Major keys are associated with topics like "captain ship," while minor keys are linked to themes of separation or FBI agents.
- Acousticness is strongly associated with topics like "mr. sir mrs." (period pieces, older films).
- Danceability is linked to topics like "baby yo" (urban films) and "women married" (sitcoms).
- Fast tempos correlate with topics like "dude cool" (comedies), while slow tempos are linked to "mr. sir mrs."
- High instrumentalness is associated with topics like "sir colonel" (sci-fi) and classical music.
- Impact on Ratings: After controlling for topic and genre using coarsened exact matching:
- Danceability has a small negative impact (e.g., a film rating drops from 9.0 to 8.897).
- Acousticness has a small positive impact (raises rating to 9.121).
- Instrumentalness has a small positive impact (raises rating to 9.117).
- Tempo showed no significant impact on ratings in this analysis.
Limitations and Future Work
- Limitations:
- The analysis of music-soundtrack relationships is based on the entirety of the film and soundtrack, lacking fine-grained temporal anchoring of individual songs to specific moments in the narrative.
- The causal inference model is acknowledged as potentially mis-specified, as other unmeasured film content factors might influence ratings and be correlated with audio features.
- Future Work:
- Develop more fine-grained models that can anchor song occurrences to specific moments in the temporal narrative of a script.
- Explore soundtrack generation: creating new soundtracks for films given a script as input, which could be a powerful tool for music supervisors.
- Investigate the role of music in other multimodal storytelling forms like television, advertising, social media, and games.
Summary updated