Abstract
Recent studies have shown that for subjective annotation tasks, the demographics, lived experiences, and identity of annotators can have a large impact on how items are labeled. We expand on this work, hypothesizing that gender may correlate with differences in annotations for a number of NLP benchmarks, including those that are fairly subjective (e.g., affect in text) and those that are typically considered to be objective (e.g., natural language inference). We develop a robust framework to test for differences in annotation across genders for four benchmark datasets. While our results largely show a lack of statistically significant differences in annotation by males and females for these tasks, the framework can be used to analyze differences in annotation between various other demographic groups in future work. Finally, we note that most datasets are collected without annotator demographics and released only in aggregate form; we call on the community to consider annotator demographics as data is collected, and to release dis-aggregated data to allow for further work analyzing variability among annotators.
Introduction: The Myth of the Objective "Ground Truth"
In Natural Language Processing (NLP), researchers often treat "ground truth" labels as objective facts. However, human language is inherently subjective. Previous research has demonstrated that an annotator’s identity—including culture, age, and gender—can significantly influence how they label data, particularly for sensitive topics like hate speech.
This study investigates whether "annotator bias" extends to broader NLP benchmarks. Does gender affect how people rate the "joy" in a news headline or whether one sentence logically follows another? By identifying these differences, researchers can build AI systems that are more inclusive of diverse perspectives rather than forcing a single, potentially biased "average" view.
Methodology: A Framework for Demographic Analysis
The researchers developed a two-pronged statistical framework to analyze four different NLP datasets:
- Distribution Analysis: They compared the "spread" of scores given by men versus women. For continuous data (e.g., a scale of 0-100), they used Kernel Density Estimation (KDE) plots. For discrete categories, they used bar plots. To ensure observed differences weren't due to chance, they employed permutation testing with 10,000 iterations.
- Agreement Analysis: They measured how much individual men and women agreed with the "aggregate" (the average score of all other annotators). They used Krippendorff’s alpha, a versatile metric that handles various data scales, to determine if one gender was more "in sync" with the majority.
Datasets and Experimental Setup
The study utilized four datasets that provided both individual (non-aggregated) annotations and the gender of the annotators:
- Affective Text: Headlines rated for six emotions (anger, disgust, fear, joy, sadness, surprise) and valence.
- Word Similarity: Pairs of words rated on their similarity or relatedness.
- Sentiment Analysis: Blog posts about aging rated on a 5-point scale from "very negative" to "very positive."
- Natural Language Inference (NLI): The CommitmentBank dataset, where annotators judge a speaker's commitment to a statement.
Results: Where Gender Matters (and Where It Doesn't)
The findings provide a nuanced view of gender's role in annotation:
- Sentiment Analysis: This was the only task with a statistically significant difference ($p = 0.0209$). Men were found to use more "middle-ground" or intermediary labels (e.g., "somewhat" positive/negative), whereas women were more likely to choose "extreme" or "neutral" labels.
- Word Similarity: A slight trend was observed ($p = 0.0528$) where men gave generally higher similarity scores, though this did not meet the threshold for statistical significance after multiple-test corrections.
- Agreement Levels: No significant difference was found in how much men versus women agreed with the group average. Neither gender proved to be "more consistent" or "better" at these tasks.
- Affective Text: While visual plots showed some spikes, permutation tests revealed these were likely due to a few "extreme" individual annotators rather than a broader gender-wide trend.
Discussion and Future Directions
The study highlights several critical points for the NLP community. First, the "negative result"—finding no significant gender difference in most tasks—is valuable for preventing researchers from pursuing non-existent biases. Second, the subjectivity observed even in "objective" tasks like NLI appears to be individual rather than systematically tied to gender.
However, the work faced significant limitations. The study was constrained to a male/female binary due to existing data collection methods, and small sample sizes in some datasets (e.g., only six annotators for Affective Text) limited statistical power. Furthermore, gender does not exist in a vacuum; future research should explore intersectionality—how gender interacts with age, race, and geography—using tools like Generalized Additive Models (GAMs).
The authors conclude with a strong call to action: the NLP community must prioritize collecting and releasing "disaggregated" data. Transparency regarding individual votes is essential for auditing AI models for demographic bias and ensuring they represent a truly diverse human perspective.
Summary updated