Voices in a Crowd: Searching for clusters of unique perspectives

Nikolas Vitsakis, Amit Parekh, Ioannis KonstasEMNLP 2024 · DOI

Introduction

In Natural Language Processing (NLP), the standard paradigm of supervised learning relies on the assumption of a "single ground truth" label for every data point. However, for subjective tasks such as detecting political bias, toxic language, or sentiment, human annotators frequently disagree. This disagreement is not merely noise; it often reflects valid, diverse perspectives rooted in different backgrounds and beliefs.

Annotator disagreement on subjective climate change and media bias examples.

Existing approaches to handle this disagreement typically fall into two categories:

  1. Disagreement-based models treat labels as a distribution, which often collapses nuanced minority views into a single "non-majority" category.
  2. Metadata-based models group annotators by demographic attributes (e.g., gender, political leaning), which can lead to reductive stereotyping and fails to capture the true diversity of thought within those groups.

This paper introduces "Voices in a Crowd," a framework designed to identify distinct "voices"—groups of people sharing a perspective—based on their actual labeling behavior rather than pre-defined demographic categories.

Methodology: Behavioral Embeddings and Clustering

The proposed framework utilizes a metadata-agnostic approach to discover latent perspectives within a crowd of annotators. The pipeline consists of three primary stages:

  1. Supervised Prediction: A model is trained to predict individual annotations for every unique annotator. The researchers found that an Encoder-Encoder architecture (using a T5 backbone) was most effective. This architecture allows the text and the unique annotator ID to interact via bidirectional attention, capturing how specific individuals react to specific content.
  2. Behavioral Embedding Extraction: Instead of relying solely on the final label prediction, the framework extracts the model's final hidden states. These "behavioral embeddings" represent the latent logic or "voice" of an annotator's response to a given text.
  3. Unsupervised Clustering: These embeddings are then grouped using dimensionality reduction (e.g., UMAP) and clustering algorithms (e.g., K-means).
Comparison of the "Voices in a Crowd" framework against traditional disagreement-based and metadata-based approaches.

Crucially, the model remains metadata-agnostic during training. Demographic information is only used post-hoc to validate that the discovered clusters correspond to meaningful social or political groups.

Experimental Setup and Results

The framework was evaluated on two politically charged datasets:

  • Media Bias Annotation Dataset (MBIC): 784 annotators labeling news sentences for bias.
  • Global Warming Stance Dataset (GWSD): 398 annotators labeling climate change statements.

Key Findings

  • Predictive Performance: The Encoder-Encoder model achieved F1 scores of 0.72 on MBIC and 0.63 on GWSD, outperforming simpler baseline architectures.
  • Cluster Robustness: The combination of UMAP and K-means produced the most robust clusters. On the MBIC dataset, the model achieved a Silhouette score of 0.53, indicating strong separation between the identified "voices."
  • Richness of Signal: Qualitative analysis revealed that behavioral clusters often captured "inter-minority" perspectives—such as highly educated right-leaning individuals—that are typically lost in broader demographic-based models.

Discussion and Future Work

The "Voices in a Crowd" framework demonstrates that annotator behavior is a far richer signal for modeling subjectivity than static demographic metadata. By focusing on how people actually label data, we can move toward NLP systems that respect and represent a plurality of perspectives.

Future research directions include addressing annotator sparsity, as the model requires enough labels per individual to form a stable profile. Additionally, integrating Large Language Models (LLMs) to automatically interpret and describe the perspective of each cluster could further automate the discovery of unique "voices" in large-scale datasets.

Summary updated

BibTeX
@inproceedings{vitsakis2024voices,
  author = {Nikolas Vitsakis and Amit Parekh and Ioannis Konstas},
  title = {{Voices in a Crowd: Searching for clusters of unique perspectives}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.696},
}
Figure