TL;DR
This paper introduces GRASP, a novel framework for analyzing systematic disagreements in human annotations across different demographic groups. By quantifying in-group and cross-group cohesion, GRASP helps identify which groups hold significantly different perspectives on subjective tasks, offering a more nuanced understanding of annotation diversity and its implications for AI development.
What Problem Does This Paper Address?
Human annotation is crucial for training and evaluating AI models, especially for tasks involving subjective judgments like safety and offensiveness. However, existing methods often overlook the inherent subjectivity of human raters, leading to a "one-size-fits-all" approach that can marginalize minority perspectives and amplify societal biases. While previous work has hinted at demographic differences in annotations, there's a lack of generalizable methods to systematically measure and understand these group-level associations in perspectives. This paper aims to fill that gap by providing a framework to quantify and statistically validate these differences.
Methods
The core of the GRASP framework lies in two key concepts:
- In-group Cohesion ($C_I$): Measures how much raters within a specific demographic group agree with each other.
- Cross-group Cohesion ($C_X$): Measures how much one rater group agrees with another group.
These cohesion metrics are combined into a Group Association Index (GAI), which quantifies the strength of a demographic group's association with a particular annotation pattern. A GAI greater than 1 suggests higher agreement within the group compared to outside, indicating a distinct perspective.
To ensure the reliability of these measurements, GRASP employs permutation tests for statistical significance testing. This approach accounts for the dependencies within the data, which is crucial for annotation datasets. The framework is designed to be flexible, allowing for various underlying metrics for cohesion and divergence, and can analyze groups defined by any combination of demographic attributes.
Datasets
The GRASP framework was applied to two existing datasets:
- DICES-350: A dataset of 350 chatbot conversations annotated for safety by 104 raters. This dataset was specifically curated to have a diverse pool of raters across age, gender, and race.
- D3: A dataset of social media posts annotated for offensiveness by 4000 raters, balanced across cultural regions, gender, and age.
Key Insights and Ideas
- Subjectivity is Systematic: Disagreements in human annotations aren't just random noise; they can be systematic and tied to demographic factors.
- Beyond Majority Vote: Relying solely on majority votes can obscure important minority perspectives and reinforce existing biases. GRASP offers a way to quantify these differences.
- Demographics Matter, But How?: The framework helps identify which demographic axes (e.g., race, gender, age, or intersections thereof) are most influential for specific tasks.
- Task-Dependent Associations: The influence of demographic factors on annotation perspectives varies significantly depending on the task (e.g., safety vs. offensiveness).
- Intersectional Analysis is Key: Analyzing demographic groups in isolation is insufficient; understanding intersections (e.g., Latine women) reveals more nuanced patterns of agreement and disagreement.
Key Results
The application of GRASP to DICES-350 and D3 revealed several key findings:
- DICES-350 (Safety Annotations):
- Race/ethnicity showed significant results on its own, suggesting age and gender were less influential individually.
- Intersectional analysis highlighted Latine women as having the highest in-group cohesion (0.238).
- White men had the highest GAI (1.262), indicating a strong association with a particular perspective, while Asian women had the lowest GAI (0.540), suggesting more agreement with raters outside their group.
- D3 (Offensiveness Annotations):
- Age and region groups showed significant associations.
- 50+-year-old raters from Sinosphere had the highest GAI (2.225), followed by non-binary genders (2.172).
- Intersectional results showed that while women generally didn't report high GAI, specific subgroups of women in different regions exhibited more in-group agreement.
- The study also found that different metrics (IRR, Negentropy, Plurality size) capture different aspects of agreement and can yield varying results, emphasizing the need to choose metrics carefully based on the task.
Limitations and Future Work
- Demographic Simplification: The demographic breakdowns in the datasets are simplified representations of the population. More granular data could reveal further insights.
- Other Intersectional Groups: The paper focused on specific intersectional groups; further research could explore other combinations.
- Distinguishing "Good" vs. "Bad" Disagreement: While GRASP identifies systematic disagreements, further work is needed to disentangle disagreements that represent valuable diverse perspectives from those that might indicate issues like lack of training or rater bias.
- Metric Selection: The paper notes that the choice of metric can influence results, and future work should investigate which metrics are most suitable for different tasks and data characteristics.
- Dynamic Data Collection: The framework could be extended to enable dynamic data collection that adapts to emergent group associations.
- Understanding Reasons for Disagreement: Further investigation is needed to understand the underlying reasons (e.g., individual moral values) behind the observed systematic disagreements.
Summary updated