TL;DR
This paper introduces a computational approach to automate the Bechdel test, a measure of female representation in film. By analyzing movie scripts, they found that movies failing the test indeed portray women as less central and important characters, with social network analysis features proving most effective in identifying this.
What Problem Does This Paper Address?
The Bechdel test, a simple set of three questions, is widely used as a proxy for assessing gender bias in media. However, manually applying it to a large corpus of films is time-consuming and subjective. This paper addresses the need for an automated method to apply the Bechdel test, enabling large-scale analysis of gender representation in film and its correlation with character importance. The authors aim to computationally determine if movies pass or fail the Bechdel test and to investigate the linguistic and social network characteristics that differentiate passing from failing films.
Methods
The core of this work involves developing computational models to automate the three Bechdel test questions:
- Test 1: Are there at least two named women in the movie? This was tackled by using various name-to-gender resources (IMDB_GMAP, SSA_GMAP) and a novel technique called STAN_GMAP, which uses coreference resolution on scene descriptions to infer gender from pronouns.
- Test 2: Do these women talk to each other? This was addressed by constructing interaction networks between characters within scenes. Two approaches were explored: CLIQUE (connecting all characters within a scene boundary) and CONSECUTIVE (connecting characters who speak consecutively).
- Test 3: Do these women talk to each other about something besides a man? This was framed as a machine learning task, utilizing a combination of linguistic features (word unigrams, topic models, dialogue analysis) and social network analysis (SNA) features derived from character interaction networks.
The authors experimented with different classifiers, including Logistic Regression and Support Vector Machines (SVM) with linear and RBF kernels, and employed a weighted loss function to penalize misclassifications of the minority class (movies failing the test).
Datasets
The primary dataset used in this study consists of movie screenplays.
- IMSDB: The Internet Movie Script Database was crawled, yielding 964 movie screenplays.
- bechdeltest.com: Labels for 457 movies were obtained from this website, which has user-submitted ratings for the Bechdel test.
- Training/Development Set: 367 movies were used for training and development.
- Test Set: 90 movies were reserved for testing.
Key Insights and Ideas
- Social Network Analysis is Key for Test 3: The most surprising finding is that social network analysis features, particularly those related to character centrality and connectivity, are significantly more effective than linguistic features for determining if women talk about topics other than men. This suggests that the structure of interactions, rather than just the words spoken, is crucial for this aspect of the Bechdel test.
- Failing Movies = Less Central Women: Movies that fail the Bechdel test are characterized by women who are significantly less centrally connected in the narrative's social network. This provides empirical support for the intuition that failing the Bechdel test correlates with women having less important or peripheral roles.
- Gender Identification Challenges: Accurately identifying character genders is a foundational challenge. The authors found that combining multiple resources (STAN_GMAP + IMDB_GMAP) yielded better results than individual ones, highlighting the need for robust gender-assignment methods.
- Context Matters for Dialogue Analysis: The CONSECUTIVE approach for interaction networks outperformed the CLIQUE approach, indicating that the immediate sequence of dialogue is more informative than simply grouping all characters in a scene.
Key Results
- Test 1 Performance: The combined STAN_GMAP and IMDB_GMAP approach achieved a macro-F1 score of 0.75 for Test 1, outperforming individual resources.
- Test 2 Performance: The CONSECUTIVE approach for interaction networks achieved a macro-F1 score of 0.62, outperforming the CLIQUE approach.
- Test 3 Performance: Social network analysis (SNA) features significantly outperformed linguistic features (BOW, LING, TOPIC) and the baseline features proposed by Garcia et al. (2014), achieving a macro-F1 score of 0.68.
- End Task Performance: The best feature set and classifier combination for the overall Bechdel test achieved a macro-F1 score of 0.80 on the unseen test set, outperforming the baseline by a significant margin.
Limitations and Future Work
- Limitations:
- The accuracy of gender identification is a known limitation, especially with ambiguous names or non-binary characters.
- The coreference resolution system was applied only to scene descriptions, potentially missing crucial pronoun references within dialogues.
- Inconsistent screenplay formatting and anomalies in web-scraped data posed challenges.
- The "dubious" flag on bechdeltest.com indicates subjectivity in manual ratings, which can impact the gold standard.
- Future Work:
- The authors suggest extending this work to analyze children's books and novels, applying similar social network extraction techniques.
- Further research could explore more sophisticated linguistic and social network features.
- Investigating the correlation between Bechdel test results and other measures of gender representation in media is also a promising direction.
Summary updated