In traditional machine learning, human disagreement is often treated as "noise." When three annotators label a tweet as toxic and two do not, the standard procedure is to take a majority vote and discard the minority opinion. This "ground truth" paradigm assumes there is a single correct answer for every input.
However, for subjective tasks like hate speech detection, humor, or aesthetic judgment, disagreement is not noise—it is a signal. Different people bring different sociocultural backgrounds, biases, and sensitivities to their work. Recent research has shifted toward perspectivist NLP, where the goal is to model the individual perspectives of annotators rather than collapsing them into a single average.
The Annotator as a First-Class Feature
The core intuition behind this research direction is that a model should not just learn $P(\text{label} \mid \text{item})$, but rather $P(\text{label} \mid \text{item}, \text{annotator})$. By treating the annotator as a primary input, models can learn to predict how a specific person—or a specific type of person—will perceive a piece of content.
Emmanouil Antonios Platanios and colleagues in Learning from Imperfect Annotations demonstrated that by jointly learning the ground truth, the difficulty of the instance, and the competence of the annotator, models can achieve significant accuracy gains while requiring fewer redundant annotations. Similarly, the DisCo framework introduced by Tharindu Cyril Weerasooriya in Disagreement Matters preserves label diversity by modeling the full distribution of human opinions, ensuring that minority perspectives are not erased.
Representing Human Perspective via Latent Embeddings
To make an annotator "readable" by a neural network, researchers represent them as latent embeddings—continuous vectors in a high-dimensional space. These embeddings capture idiosyncratic labeling patterns, biases, and tendencies.
There are several ways to construct these representations:
- User Tokens: The simplest approach, explored by Olufunke Oluyemi Sarumi in Corpus Considerations for Annotator Modeling and Scaling, involves appending a unique, learnable identification token to the text input. This allows the model to associate specific biases with a discrete ID.
- Label-Centric Embeddings: Naihao Deng and Lu Wang in You Are What You Annotate propose calculating an annotator's "style" by averaging the embeddings of all labels they have assigned. This captures whether an annotator is generally lenient or severe.
- Metadata and Sociocultural Descriptors: Mohammed Fayiz Parappan in Learning Subjective Label Distributions via Sociocultural Descriptors incorporates demographic data like age, gender, and locale into the embeddings to condition predictions on cultural context.
Architectural Integration: Gating and Queries
Once we have an annotator representation, how does the model use it? The goal is to "condition" the model's internal logic on the annotator's perspective.
In the AVFS (A View From Somewhere) framework, Jerone Andrews and Przemyslaw Joniak use annotator-specific gating functions in A View From Somewhere: Human-Centric Face Representations. These gates weight different dimensions of the visual embedding space, reflecting how different people prioritize different features (e.g., one person might judge face similarity based on eyes, while another focuses on face shape).
Alternatively, the QuMATL framework by Liyun Zhang and Hong Liu uses learnable query vectors within a Transformer architecture. These queries allow the model to "ask" how a specific annotator's tendency interacts with the input text, providing a lightweight way to personalize predictions without training a separate model for every user.
Beyond Demographics: Behavioral Clustering
A common pitfall in this area is relying too heavily on demographic metadata (like "age" or "gender"), which can be reductive or unavailable. Nikolas Vitsakis and Ioannis Konstas address this in Voices in a Crowd by extracting behavioral embeddings from the hidden states of models trained to predict individual labels.
By clustering these embeddings, they can identify distinct "voices" or schools of thought within a dataset based purely on how people label data, rather than who they are. This approach is robust even when metadata is missing and can reveal nuanced minority perspectives that demographic categories might miss. Similarly, Marek Herde and Bernhard Sick use latent embeddings in the MaDL framework to identify and neutralize clusters of low-quality "spammers" who exhibit correlated error patterns.
Why It Works: Personalization over Generalization
The shift toward annotator modeling works because it acknowledges that "subjectivity" is not random error—it is structured variation. As Matthias Orlikowski and Dirk Hovy argue in Beyond Demographics, Large Language Models are often better at "memorizing" the idiosyncratic patterns of specific individuals than they are at learning broad social group trends.
By providing the model with a unique identifier or a behavioral history—as seen in the work on toxicity prediction by Harbani Jaggi and Eve Fleisig—we allow the model to resolve ambiguity. What looks like a contradiction in a "majority vote" dataset becomes a clear, predictable pattern when viewed through the lens of individual annotator embeddings.