Learning from Sparse and Subjective Human Annotations

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 1140 papers180 conceptsfrozen 2026-10-04

In the traditional supervised learning paradigm, we treat human labels as "ground truth." If three annotators look at a picture of a cat and two say "cat" while one says "dog," we simply take the majority vote and move on. However, this approach breaks down in two critical scenarios: when the task is inherently subjective (like identifying hate speech or sentiment) and when the data is extremely sparse (where most items receive only one or two labels).

A growing body of research suggests that human disagreement is not just "noise" to be filtered out, but a vital signal. By modeling the unique perspectives of individual annotators and leveraging the latent relationships between items, researchers are developing ways to build more robust, nuanced AI even when data is thin and opinions vary.

The Challenge of Sparse Disagreement

The fundamental problem in modern crowdsourcing is a lack of overlap. In a perfect world, every annotator would label every item, creating a dense matrix of opinions. In reality, budget and time constraints mean that individual workers only see a tiny fraction of the total dataset. As Shao-Yuan Li and colleagues note in their work on DASC, this sparsity makes it nearly impossible to use standard statistical metrics like Cohen’s Kappa to establish reliability.

When an item has only one label, we cannot know if that label represents a broad consensus or a personal outlier. This "construct gap," as described by Maria De-Arteaga and Alexandra Chouldechova, creates a mismatch between the proxy labels we have and the complex reality we want the model to predict.

Inferring Truth through Latent Relationships

To overcome this sparsity, researchers have turned to techniques borrowed from recommendation systems. If Netflix can predict which movie you’ll like based on a few ratings, can we predict how an annotator would label a sentence based on a few previous judgments?

Hyun Joon Jung and Matthew Lease pioneered the use of Probabilistic Matrix Factorization to fill in these missing labels. By treating annotators as "users" and tasks as "items," they can infer a worker's latent tendencies. Similarly, the MiSC framework by Ching Yun Ko uses low-rank tensor completion to reconstruct the full landscape of human opinion from a fragmented set of observations.

The intuition here is that annotator errors and biases are rarely random. An expert might be highly reliable for general categories but struggle with fine-grained distinctions. By modeling these "sample-wise" reliabilities—as Zhengqi Gao and Fan-Keng Sun do in their label fusion framework—models can learn to weigh different voices differently depending on the specific context of the data point.

From Majority Rule to Soft Distributions

For subjective tasks, the goal is shifting from finding a single "correct" label to predicting the full distribution of human perspectives. Alexandra Uma and her colleagues argue for the use of Soft Loss Functions, showing that training on the full spectrum of crowd judgments helps models capture inherent ambiguity and generalize better to unseen data.

Models like DisCo (Distribution from Context), developed by Tharindu Cyril Weerasooriya and Christopher M. Homan, explicitly preserve minority viewpoints. Instead of erasing a lone dissenting voice as "noise," these models treat that dissent as a valid part of the human response distribution. This is particularly crucial in fields like hate speech detection, where what one group finds offensive might be perceived differently by another.

Scaling Subjectivity with Artificial Annotators

As we move toward even larger datasets, the cost of collecting enough human labels to form a distribution becomes prohibitive. To solve this, researchers are experimenting with "Artificial Annotators"—ensembles of models trained to simulate different modes of human opinion. Xinliang Frederick Zhang and Marie-Catherine de Marneffe demonstrated that these simulated viewpoints can successfully distinguish between clear-cut facts and items prone to inherent disagreement.

More recently, Yuxiang Zhou and Maria Liakata have shown that injecting personal profiles (like personality traits and demographics) into Large Language Models allows them to better simulate the subjective variance of human cognitive appraisal.

Why It Works: The Power of Shared Prototypes

The most effective recent models, such as PTBCC by Ju Chen, address the sparsity problem by assuming that while every annotator is unique, they often share common patterns of behavior. By modeling annotators as a mixture of shared "prototypes," these systems can accurately predict the behavior of a new worker who has only provided a handful of labels.

By moving away from the rigid "gold standard" and embracing the complexity of human disagreement, this research direction is creating AI that is not only more accurate in sparse environments but also more representative of the diverse world it is meant to model.

Papers in this version

Showing 60 of 140 papers.

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_learning_from_sparse_and_subjective_huma_959df1c0,
  title = {Learning from Sparse and Subjective Human Annotations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/learning-from-sparse-and-subjective-human-annotations/rdr_959df1c0911c4579a28fb6b2c269d12f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_learning_from_sparse_and_subjective_huma_959df1c0_v1,
  title = {Learning from Sparse and Subjective Human Annotations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/learning-from-sparse-and-subjective-human-annotations/rdr_959df1c0911c4579a28fb6b2c269d12f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}