Introduction
In Natural Language Processing (NLP), the standard paradigm of supervised learning relies on the assumption of a "single ground truth" label for every data point. However, for subjective tasks such as detecting political bias, toxic language, or sentiment, human annotators frequently disagree. This disagreement is not merely noise; it often reflects valid, diverse perspectives rooted in different backgrounds and beliefs.
Existing approaches to handle this disagreement typically fall into two categories:
- Disagreement-based models treat labels as a distribution, which often collapses nuanced minority views into a single "non-majority" category.
- Metadata-based models group annotators by demographic attributes (e.g., gender, political leaning), which can lead to reductive stereotyping and fails to capture the true diversity of thought within those groups.
This paper introduces "Voices in a Crowd," a framework designed to identify distinct "voices"—groups of people sharing a perspective—based on their actual labeling behavior rather than pre-defined demographic categories.
Methodology: Behavioral Embeddings and Clustering
The proposed framework utilizes a metadata-agnostic approach to discover latent perspectives within a crowd of annotators. The pipeline consists of three primary stages:
- Supervised Prediction: A model is trained to predict individual annotations for every unique annotator. The researchers found that an Encoder-Encoder architecture (using a T5 backbone) was most effective. This architecture allows the text and the unique annotator ID to interact via bidirectional attention, capturing how specific individuals react to specific content.
- Behavioral Embedding Extraction: Instead of relying solely on the final label prediction, the framework extracts the model's final hidden states. These "behavioral embeddings" represent the latent logic or "voice" of an annotator's response to a given text.
- Unsupervised Clustering: These embeddings are then grouped using dimensionality reduction (e.g., UMAP) and clustering algorithms (e.g., K-means).
Crucially, the model remains metadata-agnostic during training. Demographic information is only used post-hoc to validate that the discovered clusters correspond to meaningful social or political groups.
Experimental Setup and Results
The framework was evaluated on two politically charged datasets:
- Media Bias Annotation Dataset (MBIC): 784 annotators labeling news sentences for bias.
- Global Warming Stance Dataset (GWSD): 398 annotators labeling climate change statements.
Key Findings
- Predictive Performance: The Encoder-Encoder model achieved F1 scores of 0.72 on MBIC and 0.63 on GWSD, outperforming simpler baseline architectures.
- Cluster Robustness: The combination of UMAP and K-means produced the most robust clusters. On the MBIC dataset, the model achieved a Silhouette score of 0.53, indicating strong separation between the identified "voices."
- Richness of Signal: Qualitative analysis revealed that behavioral clusters often captured "inter-minority" perspectives—such as highly educated right-leaning individuals—that are typically lost in broader demographic-based models.
Discussion and Future Work
The "Voices in a Crowd" framework demonstrates that annotator behavior is a far richer signal for modeling subjectivity than static demographic metadata. By focusing on how people actually label data, we can move toward NLP systems that respect and represent a plurality of perspectives.
Future research directions include addressing annotator sparsity, as the model requires enough labels per individual to form a stable profile. Additionally, integrating Large Language Models (LLMs) to automatically interpret and describe the perspective of each cluster could further automate the discovery of unique "voices" in large-scale datasets.
Summary updated