TL;DR
This paper tackles the challenge of detecting sexism in sitcom dialogues, a domain often lacking large annotated datasets. They introduce a novel semi-supervised approach that leverages existing tweet-based sexism datasets to build a more robust classifier for sitcom scripts, demonstrating improved performance over generic models.
What Problem Does This Paper Address?
Television sitcoms, due to their popularity and influence, play a significant role in shaping societal norms and attitudes, particularly among younger audiences. While documenting sexism in media is crucial for understanding social progress, creating comprehensive datasets for this purpose is challenging. Manual annotation of dialogue is time-consuming, expensive, and prone to annotator bias. Existing datasets for sexism detection are often focused on social media (like tweets), which have distinct linguistic characteristics (short, informal, slang-filled) compared to the more descriptive and context-dependent dialogue found in sitcoms. This paper aims to bridge this gap by developing a machine learning model capable of accurately identifying sexism in sitcom dialogue, even with limited labeled data.
Methods
The core of this paper's methodology lies in domain adaptation and semi-supervised learning. They start with a pre-existing dataset of sexist and neutral tweets (from Waseem and Hovy, 2016) as their "source domain." To adapt this to the "target domain" of sitcom dialogue, they employ a semi-supervised approach.
- Initial Model Training: They first train several standard text classification models (NB, RF, LR, SVM, BERT, Bi-LSTM) on a combined dataset of tweets and a small, manually annotated set of sitcom dialogues (Dtrain). This helps identify the most effective architecture for the task.
- Pseudo-labeling: The best-performing model (Bi-LSTM) is then used to predict sexism labels on a large corpus of unannotated sitcom dialogues (Dsemisupervised). This process generates "pseudo-labels" for the unlabeled data.
- Dataset Augmentation: The pseudo-labeled sitcom dialogues are combined with the original manually annotated sitcom dialogues and the tweet dataset to create a larger, augmented training set (Dfinal).
- Final Model Training: A Bi-LSTM model is then trained on this augmented Dfinal dataset. This approach allows the model to learn from a much larger, albeit partially noisy, dataset, improving its generalization to the sitcom domain.
Datasets
- Source Domain (Tweets): A dataset of sexist and neutral tweets from Waseem and Hovy (2016).
- Target Domain (Sitcom Dialogues):
- Dtrain: A small, manually annotated dataset of sitcom dialogue turns, comprising 203 sexist and 926 neutral turns. This was created by manually annotating 1,937 dialogue turns from twenty popular sitcoms.
- Dsemisupervised: A large collection of unannotated dialogue turns from the same twenty sitcoms, used for pseudo-labeling.
- Dtest: A held-out, manually annotated test set of 805 dialogue turns (411 neutral, 394 sexist), annotated by four annotators with a Cohen's Kappa of 0.87.
- Dfinal: The augmented training dataset, combining Dtrain, Dsemisupervised (pseudo-labeled), and the tweet dataset.
Key Insights and Ideas
- Domain Adaptation is Crucial: The significant difference in linguistic style between tweets and sitcom dialogue necessitates domain-specific adaptation. A model trained solely on tweets would likely perform poorly on sitcoms, and vice-versa.
- Semi-Supervised Learning Bridges the Data Gap: By using a strong base model to generate pseudo-labels on unlabeled data, the authors effectively expand their training set, mitigating the challenge of limited manually annotated data.
- Bi-LSTM Excels for This Task: The Bi-LSTM architecture proved to be the most effective for capturing the sequential and contextual nature of dialogue, outperforming other models like BERT and SVM.
- Sexism Trends are Nuanced: While overall sexism in sitcoms appears to be decreasing over time, the proportion of sexist dialogue in the most sexist shows has actually increased, suggesting a complex evolution rather than a simple linear decline.
Key Results
- The proposed semi-supervised Bi-LSTM model trained on Dfinal achieved an accuracy of 83.0% on the held-out test set (Dtest), a significant improvement over the baseline Bi-LSTM trained only on Dtrain (76.03%).
- Other models also showed an average improvement of 4.67% when trained on the augmented Dfinal dataset compared to Dtrain.
- Content Analysis:
- Between 1985-1999, the average percentage of sexist dialogue turns was 2.26%.
- Between 2000-2015, the average decreased to 1.87%.
- However, the highest percentage of sexist dialogue turns in older shows was 2.61%, while in newer shows, the worst offender had 4.13%.
- Error Analysis: The model struggled with nuanced statements, particularly those involving:
- Aggressively negative statements about a specific woman.
- Statements containing sexual terms, which can be misclassified due to overlap with sexist content in the training data.
- Dialogues related to marriages or weddings, possibly due to insufficient contextual examples.
- The use of the pronoun "she" referring to inanimate objects.
Limitations and Future Work
- Limitations:
- The model's inability to fully grasp nuanced sexism, especially in cases of subtle discrimination or when gendered pronouns are used for objects.
- The reliance on pseudo-labels, which can introduce noise and errors into the training data.
- The definition of sexism used is based on established frameworks but might not capture all evolving forms of subtle discrimination.
- Future Work: The authors suggest further exploration into improving the model's ability to handle nuanced sexism and potentially expanding the dataset to include more diverse forms of media or more granular annotation categories. The current work provides a strong foundation for tracking social discrimination in media through automated analysis.
Summary updated