TL;DR
Researchers at Cornell University developed a statistical language-model-based approach to quantify gender bias in sports journalism by measuring how "game-related" interview questions are. Analyzing over 80,000 tennis interview snippets, they found that journalists ask male athletes significantly more game-focused questions, a bias that becomes even more pronounced during "off-the-wall" queries and for lower-ranked female players.
What Problem Does This Paper Address?
Gender bias in sports media is a well-documented but often qualitatively described phenomenon. While initiatives like Cover the Athlete highlight inappropriate or sexist questioning of female athletes, academic studies have sometimes yielded conflicting results—some manual analyses even suggested male athletes receive more descriptors regarding their physical appearance.
The challenge lies in the methodology: prior investigations relied heavily on manual coding or predefined keyword lists, which are difficult to scale and prone to subjective bias. This paper addresses the need for a quantitative, data-driven framework to measure how journalistic focus shifts depending on the athlete's gender, specifically focusing on the "game-relatedness" of questions.
Methods
The authors propose a novel application of Perplexity (PP), a standard NLP metric used to evaluate how well a language model predicts a sample.
- Modeling "Game Language": They trained a bigram language model (using KenLM) on a gender-balanced set of live play-by-play tennis commentaries. This model represents the "gold standard" for pure, game-focused language.
- Quantifying Bias: For any interview question, they calculated its perplexity relative to this game-language model.
- Low Perplexity: The question uses language similar to match commentary (e.g., "What about your serve?").
- High Perplexity: The question deviates significantly from match commentary, indicating a focus on off-court topics (e.g., "Who designed your clothes?").
- Atypicality Scoring: To distinguish between standard press-conference fodder and "off-the-wall" queries, they used Inverse Document Frequency (IDF) to calculate an "atypicality score" for each question.
Datasets
The study utilized three primary data sources:
- ASAP Sports: 6,467 tennis press-conference transcripts (2000–2015), totaling 81,906 question snippets posed to 167 female and 191 male players.
- Tennis-Data.co.uk: Match statistics, including player rankings and match outcomes, to provide context for the interviews.
- Sports Mole: Live text play-by-play commentaries used to train the game-language model.
The authors have released their processed dataset of tennis interview transcripts and match info to the public.
Key Insights and Ideas
- The "Game-Relatedness" Gap: Male athletes consistently receive questions with lower perplexity (more game-focused) than female athletes. This isn't just driven by a few superstars; the trend holds even when micro-averaging results per player.
- Bias in the "Off-the-Wall": While gender bias exists in "typical" questions (standard inquiries about the match), it is significantly more salient in "atypical" questions. When journalists go off-script, they are much more likely to steer away from the game when talking to women.
- The Ranking Interaction: For men, the level of game-relatedness remains relatively stable regardless of their rank. For women, however, lower-ranked players receive significantly less game-related questions than their top-10 counterparts.
- Outcome Neutrality: Interestingly, whether a player won or lost the match did not significantly impact the game-relatedness of the questions they received, suggesting the bias is systemic rather than situational.
Key Results
The study found that the mean perplexity of questions posed to male players is significantly smaller ($p < 0.001$) than those posed to females.
As shown in Figure 1, the gap in perplexity between genders widens dramatically for atypical questions. Furthermore, Figure 2 illustrates that while top-10 players of both genders receive more game-focused questions than the "rest," the gender gap persists across both tiers.
Limitations and Future Work
The authors acknowledge several limitations:
- Domain Specificity: The study is restricted to tennis. While this controls for "lingo" noise, it's unclear if these findings generalize to team sports or other individual sports.
- Journalist Anonymity: The transcripts do not identify which journalist asked which question, preventing the researchers from controlling for individual reporter bias or idiosyncrasies.
- One-Dimensional Metric: The study focuses solely on "game-relatedness." It does not account for other forms of bias, such as tone, sentiment, or specific stereotypical framing (e.g., focus on family vs. focus on physical strength).
Future work could involve training language models on other specific domains (like "personal life" or "sponsorships") to provide a more multi-dimensional view of how journalistic focus shifts.
Summary updated