No article text was recorded for this version.
probabilistic label aggregation (annotator reliability modeling)
Version 1 of 1149 papers176 conceptsfrozen 2026-10-04
Papers in this version
- Blog: Sequence labeling with multiple annotators
- Blog: Learning to Rank under Multiple Annotators
- Blog: peerannot: A framework for label aggregation in crowdsourced datasets
- Blog: Continuous-Valued Annotations Aggregation for Heart Rate Detection
- Blog: Computer Vision to the Rescue: Infant Postural Symmetry Estimation from Incongruent Annotations
- Blog: Cheap and Fast - But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks
- Blog: Inferring object rankings based on noisy pairwise comparisons from multiple annotators
- Blog: A fusocelular skin dataset with whole slide images for deep learning models
- Blog: Computer Vision to the Rescue: Infant Postural Symmetry Estimation from Incongruent Annotations
- Blog: The CrowdGleason dataset: Learning the Gleason grade from crowds and experts
- Blog: MARTA: Leveraging Human Rationales for Explainable Text Classification
- Blog: Clustering from Multiple Uncertain Experts
- Blog: An Investigation of Techniques that Aim to Improve the Quality of Labels provided by the Crowd
- Blog: Scalable Variational Gaussian Processes for Crowdsourcing: Glitch Detection in LIGO.
- Blog: Crowdsourcing for NLP
- Blog: Crowdsourced Semantic Matching of Multi-Label Annotations
- Blog: Crowdsourcing and Aggregating Nested Markable Annotations
- Blog: Answering Queries using Humans, Algorithms and Databases
- Blog: PyRater: A Python Toolkit for Annotation Analysis
- Blog: Treating Crowdsourcing as Examination: How to Score Tasks and Online Workers?
- Blog: Efficient Bayesian Inference from Noisy Pairwise Comparisons
- Blog: Aggregating and Predicting Sequence Labels from Crowd Annotations
- Blog: Annotation protocol and crowdsourcing multiple instance learning classification of skin histological images: The CR-AI4SkIN dataset
- Blog: Bayesian Detector Combination for Object Detection with Crowdsourced Annotations
- Blog: Inferring Ground Truth from Subjective Labelling of Venus Images
- Blog: Learning Whom to Trust with MACE
- Blog: Incorporating External Knowledge into Crowd Intelligence for More Specific Knowledge Acquisition
- Blog: Optimistic Knowledge Gradient Policy for Optimal Budget Allocation in Crowdsourcing
- Blog: Learning an Efficient Multi-Turn Dialogue Evaluator from Multiple Judges
- Blog: A Human-AI Loop Approach for Joint Keyword Discovery and Expectation Estimation in Micropost Event Detection
- Blog: Multi-Label Active Learning from Crowds
- Blog: Learning from Disagreement: A Survey
- Blog: Adversarial Learning from Crowds
- Blog: Learning Supervised Topic Models for Classification and Regression from Crowds
- Blog: Leveraging Crowdsourcing to Detect Improper Tasks in Crowdsourcing Marketplaces
- Blog: GPM: A Generic Probabilistic Model to Recover Annotator's Behavior and Ground Truth Labeling
- Blog: Scalable Variational Gaussian Processes for Crowdsourcing: Glitch Detection in LIGO
- Blog: Learning by aggregating experts and filtering novices: a solution to crowdsourcing problems in bioinformatics
- Blog: End-to-End Learning from Noisy Crowd to Supervised Machine Learning Models
- Blog: ConStance: Modeling Annotation Contexts to Improve Stance Classification
- Blog: Semi-Supervised Learning From Crowds Using Deep Generative Models
- Blog: Adoption and implication of the Biased-Annotator Competence Estimation (BACE) model into COVID-19 vaccine Twitter data: Human annotation for latent message features
- Blog: Multi-class Multi-annotator Active Learning with Robust Gaussian Process for Visual Recognition
- Blog: Language Understanding in the Wild: Combining Crowdsourcing and Machine Learning
- Blog: Learning from Noisy Crowd Labels with Logics
- Blog: Summary of Data Label Research under Smart Finance
- Blog: Confident in the Crowd: Bayesian Inference to Improve Data Labelling in Crowdsourcing
- Blog: On the Impossibility of Convex Inference in Human Computation
- Blog: Identifying fake Amazon reviews as learning from crowds
- Blog: FABRIC: Fully-Automated Broad Intent Categorization in E-commerce
- Blog: Dynamic Bayesian Combination of Multiple Imperfect Classifiers
- Blog: SURF: improving classifiers in production by learning from busy and noisy end users
- Blog: Sequential crowdsourced labeling as an epsilon-greedy exploration in a Markov Decision Process
- Blog: End-to-End Learning from Noisy Crowd to Supervised Machine Learning Models
- Blog: A Convex Formulation for Learning from Crowds
- Blog: Identifying the Human Values behind Arguments
- Blog: Experiments with crowdsourced re-annotation of a POS tagging data set
- Blog: Multi-winner approval voting goes epistemic
- Blog: Gaussian Process Classification and Active Learning with Multiple Annotators
- Blog: Lean Multiclass Crowdsourcing
Showing 60 of 149 papers.
History & citation
Current · Permanent link
- Modeling Annotator Reliability with Latent Confusion Matrices merged into this direction (64% overlap)
- This direction split from Probabilistic Label Aggregation for Noisy Multi-Annotator Data (51% overlap)
Cite this direction
Latest version (a living page):
@misc{lacuna_probabilistic_label_aggregation_annotato_e96eb79e,
title = {probabilistic label aggregation (annotator reliability modeling)},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/probabilistic-label-aggregation-annotator-reliability-modeling/rdr_e96eb79e0ef9457a8cdeb33bc7c1e4f0},
note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}This exact version (version 1, frozen text):
@misc{lacuna_probabilistic_label_aggregation_annotato_e96eb79e_v1,
title = {probabilistic label aggregation (annotator reliability modeling)},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/probabilistic-label-aggregation-annotator-reliability-modeling/rdr_e96eb79e0ef9457a8cdeb33bc7c1e4f0/v/1},
note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}