Learning with Probabilistic Label Distributions

17 concepts15 papers1 proposal

Standard supervised learning operates under a clean but fragile assumption: every data point belongs to exactly one ground-truth category. We represent targets as "one-hot" vectors—such as [1, 0, 0] for a three-class problem—implying absolute certainty.

However, when dealing with real-world data, this assumption quickly falls apart. If ten human experts evaluate an ambiguous medical image, or rate the emotional tone of a conversation, they rarely agree unanimously. Forcing these diverse perspectives into a single majority-vote label discards valuable information. It treats human disagreement as random noise to be filtered out, rather than an inherent feature of the data.

A growing body of research proposes a shift from discrete categories to probabilistic label distributions. Instead of forcing a single "hard" ground truth, this paradigm models labels as continuous probability distributions that explicitly capture uncertainty, ambiguity, and the variance of human opinion.


Disagreement as a Feature, Not Noise

To understand why this approach works, consider a subjective task like emotion recognition. If an audio clip sounds somewhat angry but also somewhat sad, a hard label forces an artificial choice.

In Self-Attention for Speech Emotion Recognition, Lorenzo Tarantino et al. demonstrate that utilizing label distributions that reflect annotator disagreement significantly improves model performance. By treating annotator variance as a descriptive signal rather than mere noise, the model learns to navigate emotional complexity. Similarly, Jonathan Chang shows in Learning Representations of Emotional Speech with Deep Convolutional Generative Adversarial Networks that representing target labels as probability distributions allows models to learn effectively from both the consensus and the disagreement among multiple human raters.

This perspective is central to what Eve Fleisig et al. call a "perspectivist" shift in machine learning. In The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels, the authors argue that collapsing diverse annotator opinions into a single majority vote silences minority voices and misrepresents task ambiguity. Soft labels preserve this diversity by directly mapping the frequency of each annotator's choice into the target vector.

This applies heavily to content moderation and toxicity detection, where cultural nuances dictate what is considered offensive. Xin-Qiang Cai et al. demonstrate in Soft-Label Integration for Robust Toxicity Classification that distribution-based labels allow models to better navigate the gray areas of human opinion. Rather than forcing a binary "toxic vs. non-toxic" distinction, recording confidence scores or vote distributions—as explored by Ilan Price in Six Attributes of Unhealthy Conversations—allows the training process to respect valid human disagreement in social discourse.


The Mechanics: From Hard Penalties to Distribution Matching

Why does changing the label representation fundamentally alter how a neural network learns? The answer lies in the loss function.

When training with standard Negative Log Likelihood (NLL) against a hard one-hot target, the network is heavily penalized for assigning any probability mass to incorrect classes. If a model correctly identifies a primary class but assigns a 30% probability to a highly plausible secondary class, NLL forces the network to suppress that secondary prediction, driving the model toward extreme overconfidence.

Hard Label Optimization (NLL):
Model Prediction: [0.7, 0.3]  <-- Penalized for the 0.3
Target One-Hot:   [1.0, 0.0]

Distributional Optimization (KL Divergence):
Model Prediction: [0.7, 0.3]  <-- Rewarded for matching the ambiguity
Target Soft Label: [0.6, 0.4]

By replacing point estimates with probability distributions, the training objective shifts from classification to distribution matching. In Learning an Uncertainty-Aware Object Detector for Autonomous Driving, Niranjan Thakurdesai replaces standard NLL with Kullback-Leibler (KL) divergence to train models on label distributions. This ensures that the loss function does not excessively penalize the model for being uncertain when the underlying sensor data or human annotations are genuinely ambiguous.

To formalize this, Yeshu Li et al. introduce a framework in Moment Distributionally Robust Probabilistic Supervised Learning that directly predicts conditional label distributions using distributionally robust optimization (DRO). This approach provides a much more flexible and mathematically rigorous foundation for uncertainty quantification than simply applying a softmax layer over hard categories.

This distributional framing also unlocks new data augmentation strategies. For example, the method introduced in MIDAS: Mixing Ambiguous Data with Soft Labels for Dynamic Facial Expression Recognition blends pairs of video clips alongside their corresponding soft emotion distributions. This forces the model to learn a continuous, nuanced landscape of human expressions rather than drawing arbitrary, rigid boundaries between categories.


Preventing Overconfidence and Improving Calibration

Beyond capturing subjectivity, soft label distributions solve a core structural flaw in deep neural networks: poor calibration. Modern deep networks are notorious for being wrong but highly confident.

When a model is forced to learn from ambiguous data disguised as certain data, it suffers from what the authors of TrainRef: Curating Data with Label Distribution and Minimal Reference for Accurate Prediction and Reliable Confidence call "normality pollution." Murong Ma et al. show that curating labels into distributions prevents models from becoming overconfident when encountering visually similar or inherently confusing classes, ensuring that model confidence aligns tightly with human intuition.

This representational strategy also acts as a shield against distribution shift—when the model encounters test data that looks different from its training data. In Aggregating Soft Labels from Crowd Annotations Improves Uncertainty Estimation Under Distribution Shift, Dustin Wright et al. demonstrate that simply averaging crowd-sourced annotations into soft labels preserves critical information about task difficulty. As a result, when the model faces unfamiliar out-of-distribution data, its internal uncertainty metrics remain highly reliable.

This finding is echoed in a broader data-centric context by Lars Schmarje et al. in [Is one annotation enough?

  • A data-centric image classification benchmark for noisy and ambiguous label estimation](/work/is-one-annotation-enough-a-data-centric-image). The authors show that treating ground truth as a probability distribution over classes captures aleatoric uncertainty, fundamentally improving model calibration across diverse real-world image datasets.

Application Across Modalities

The power of probabilistic label representation is its universality; it functions as an architectural-agnostic upgrade across text, audio, and vision systems.

By shifting the goal of machine learning from mimicking a forced consensus to mapping the true spectrum of human certainty, probabilistic label distributions create models that are more accurate, better calibrated, and fundamentally more honest about what they do not know.

Key Foundational Work

Learning Representations of Emotional Speech with Deep Convolutional Generative Adversarial Networks (2017) — contributed 2 concept elements to this direction.
Self-Attention for Speech Emotion Recognition (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_probabilistic_label_representation_c3bac993,
  title = {probabilistic label representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/probabilistic-label-representation/rdr_c3bac993324b45e9b47a452d9e0f859e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_probabilistic_label_representation_c3bac993_v1,
  title = {probabilistic label representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/probabilistic-label-representation/rdr_c3bac993324b45e9b47a452d9e0f859e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake