Modeling Dynamic Human Performance in Crowdsourced Labeling and Collaboration

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2183 papers248 conceptsfrozen 2026-10-04

In the standard playbook of supervised learning, we treat human annotators as "static oracles." We assume that if a worker has a 90% accuracy rate, they will maintain that 90% on the first task, the hundredth task, and the thousandth task. We treat their errors as independent, identically distributed (i.i.d.) noise—a fixed property of the individual, like their height or eye color.

However, a growing body of research suggests this "static" assumption is fundamentally flawed. Human performance is non-stationary: it fluctuates, drifts, and evolves. A worker might start a session with high precision, succumb to fatigue after an hour, or conversely, gain task-specific expertise that makes them more accurate over time. By modeling these temporal dynamics, researchers are building systems that can distinguish between genuine disagreement, lack of skill, and simple exhaustion.

The Myth of the Static Expert

When we aggregate labels from a crowd, we often use models like Dawid-Skene to estimate worker reliability. As Ofer Dekel and Ohad Shamir point out in Vox Populi: Collecting High-Quality Labels from a Crowd, these models fail to account for "temporal drift." If a worker's accuracy drops from 95% to 60% due to boredom, a static model will average these out, potentially trusting the worker too much during their "tired" phase and too little during their "expert" phase.

This non-stationarity manifests in several distinct ways:

Tracking the Moving Target

To move beyond static models, researchers are borrowing tools from time-series analysis and computational neuroscience.

One powerful approach is Visual Knowledge Tracing (VKT). Neehar Kondapaneni and Pietro Perona use recurrent neural networks to model the "non-stationary state" of a learner in Visual Knowledge Tracing. Instead of assuming a fixed internal classification function, they predict how a human’s perception of a category (like a specific bird species) shifts as they see more examples.

Similarly, Himabindu Lakkaraju and Jure Leskovec introduced Confusions over Time (CoT) in Confusions over Time: An Interpretable Bayesian Model to Characterize Trends in Decision Making. This framework audits human decisions—such as those made by judges or doctors—by modeling how their "confusion matrices" (the types of errors they make) evolve over a career. This allows for the diagnosis of systemic biases that only emerge during specific periods, such as late-afternoon fatigue or early-career inexperience.

Why Modeling Dynamics Works

The intuition behind these dynamic models is that error is not just noise; it is a signal of the annotator's state.

If we know that a worker is in a "learning" state, we can provide them with easier tasks or more feedback. If we detect a "fatigue" state, we can trigger a break or discount their labels. In Crowdsourcing with the drift diffusion model of decision making, Shamal Lalvani uses a neuroscience-inspired model to show that human performance is a dynamic accumulation of evidence, where attention levels fluctuate over time.

This perspective is particularly critical in Human-AI Collaboration. In high-stakes deferral pipelines—where an AI decides whether to handle a case or hand it to a human—the system must know the human's current reliability. As Vijay Keswani and Matthew Lease note in Designing Closed Human-in-the-loop Deferral Pipelines, if the human expert is experiencing fatigue, the AI should be less likely to defer to them, even if that human is normally highly accurate.

Beyond the Average

By embracing the "debatable" and shifting nature of human judgment, we create more robust AI. Frameworks like the one proposed by Beatriz Gomez Anson in Let's Agree to Disagree or the multitask learning approach by Uthman Jinadu in Noise Correction on Subjective Datasets suggest that we shouldn't just "clean" data to find a single truth. Instead, we should model the reasons for the noise—whether it's a temporal shift in an annotator's criteria or a genuine subjective disagreement.

The future of data collection and human-AI interaction lies in moving away from the "snapshot" view of human ability and toward a "cinematic" view that tracks the continuous, non-stationary evolution of human expertise.

Papers in this version

Showing 60 of 183 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 166 papers
  2. Version 1 · 2026-10-04 · 183 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_dynamic_worker_performance_69c65b33,
  title = {dynamic worker performance},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-worker-performance/rdr_69c65b3334f343b086d2fa31c9067b0a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_modeling_dynamic_human_performance_in_cr_69c65b33_v1,
  title = {Modeling Dynamic Human Performance in Crowdsourced Labeling and Collaboration},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/modeling-dynamic-human-performance-in-crowdsourced-labeling-and-collaboration/rdr_69c65b3334f343b086d2fa31c9067b0a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}