Traditional Natural Language Processing (NLP) operates under a clean but deeply flawed assumption: that every linguistic unit maps to exactly one discrete category. Whether determining the sense of a polysemous word, identifying the logical link between two sentences, or parsing an utterance into a semantic frame, standard benchmarks enforce a "winner-takes-all" paradigm.
However, human language is inherently fluid, vague, and multi-faceted. As Bradley Hauer et al. argue in Taxonomy of Problems in Lexical Semantics, formal semantic models that assume a word instance expresses exactly one concept fail to account for puns, metaphors, or intentional ambiguity. This single-label fallacy creates brittle systems. Samuel R. Bowman et al. point out in What Will it Take to Fix Benchmarking in Natural Language Understanding? that roughly 20% of language task examples contain legitimate linguistic ambiguity. Forcing a single gold standard label ignores this reality, causing models to overfit to annotator noise rather than capturing human-like linguistic flexibility.
Disagreement as a Signal, Not Noise
In traditional dataset curation, human annotator disagreement is treated as noise to be ironed out via majority voting or expert adjudication. A growing body of research suggests this approach discards the very signal required to model language accurately.
When multiple annotators disagree on a label, it is rarely due to random error; instead, it often reflects genuine semantic overlap. In Content and Behaviour Based Metrics for Crowd Truth, Guillermo Soberon et al. demonstrate that worker disagreement serves as a valuable indicator of text ambiguity. Similarly, Massimo Poesio et al. show in A Crowdsourced Corpus of Multiple Judgments and Disagreement on Anaphoric Interpretation that capturing multiple judgments per word reveals the natural, irreducible ambiguity of coreference.
By trying to eliminate this variance, standard pipelines misrepresent the data. As Leon Weber et al. discover in VariErr NLI: Separating Annotation Error from Human Label Variation, traditional automated error detection methods frequently mistake valid human disagreement for noise, penalizing models that recognize alternative interpretations.
Representing the Continuum: Multi-Label and Soft Frameworks
To capture this complexity, researchers are shifting from discrete classification to multi-label frameworks and soft probability distributions.
In lexical semantics, Katrin Erk et al. pioneer this shift in Graded Word Sense Assignment, treating word meaning as a continuum where multiple senses can apply to varying degrees. David Jurgens demonstrates in Embracing Ambiguity: A Comparison of Annotation Methodologies for Crowdsourcing Word Sense Labels that untrained crowd workers can produce high-quality, weighted sense labels when allowed to select multiple categories. This fluid nature is further highlighted by Rada Mihalcea et al. in SemEval-2010 Task 2: Cross-Lingual Lexical Substitution, where word meanings are modeled as overlapping sets of context-dependent translations rather than rigid dictionary definitions. By reframing Word Sense Disambiguation as a multi-label problem, Simone Conia et al. in Framing Word Sense Disambiguation as a Multi-Label Problem for Model-Agnostic Knowledge Integration allow systems to acknowledge co-occurring senses simultaneously.
A parallel evolution is occurring in discourse analysis. The logical connections between sentences are rarely mutually exclusive. The authors of DiscoGeM: A Crowdsourced Corpus of Genre-Mixed Implicit Discourse Relations show that implicit discourse relations are inherently ambiguous, supporting multiple valid semantic interpretations at once. To capture this, Sárka Zikánová et al. introduce DiscoGeM 2.0: A Parallel Corpus of English, German, French and Czech Implicit Discourse Relations, utilizing a crowdsourcing method to collect "soft labels" that represent the probability distribution of human interpretations.
Why Distributional Modeling Works
Why does moving away from single-label targets improve machine learning models? The answer lies in how models learn decision boundaries and manage uncertainty.
Standard probabilistic models optimize for a single clear peak. As Kaisheng Yao et al. note in Recurrent Support Vector Machines For Slot Tagging In Spoken Language Understanding, forcing a model to maximize the likelihood of one specific sequence fails to reflect the underlying uncertainty of ambiguous language. When a model is trained on a single ground-truth label for an inherently ambiguous event, it is penalized for making alternative, perfectly valid human-like interpretations, as shown by Mayu Otani et al. in Uncovering Hidden Challenges in Query-Based Video Moment Retrieval.
When a model is forced to choose between two highly related or co-occurring labels—such as a sentence pair that acts as both a Concession and a Temporal link—a single-label loss function forces the model to suppress one valid activation. This creates artificial gradients that penalize the model for recognizing legitimate semantic overlap.
By training models to predict a distribution of valid labels rather than a single winner, we align the loss function with linguistic reality. Alisa Liu et al. demonstrate in We're Afraid Language Models Aren't Modeling Ambiguity that multi-label frameworks enable models to flag statements that are misleading due to ambiguity, acknowledging multiple interpretations simultaneously. Furthermore, Elias Stengel-Eskin et al. show in Zero and Few-shot Semantic Parsing with Ambiguous Inputs that while zero-shot models tend to be overly "opinionated" and pick a single interpretation, few-shot models given ambiguous examples can successfully learn to match the full human distribution of meanings.
Ultimately, as Shujian Zhang et al. demonstrate in Learning with Different Amounts of Annotation: From Zero to Many Labels, allocating annotation budgets to capture the spectrum of human disagreement on a subset of data yields models that navigate real-world language uncertainty far better than those trained on rigid, single-label consensus.
Key Foundational Work
Training Conditional Random Fields Using Incomplete Annotations (2008) — contributed 1 concept elements to this direction.
Graded Word Sense Assignment (2009) — contributed 1 concept elements to this direction.