In the standard playbook of supervised learning, we treat human annotators as "static oracles." We assume that if a worker has a 90% accuracy rate, they will maintain that 90% on the first task, the hundredth task, and the thousandth task. We treat their errors as independent, identically distributed (i.i.d.) noise—a fixed property of the individual, like their height or eye color.
However, a growing body of research suggests this "static" assumption is fundamentally flawed. Human performance is non-stationary: it fluctuates, drifts, and evolves. A worker might start a session with high precision, succumb to fatigue after an hour, or conversely, gain task-specific expertise that makes them more accurate over time. By modeling these temporal dynamics, researchers are building systems that can distinguish between genuine disagreement, lack of skill, and simple exhaustion.
The Myth of the Static Expert
When we aggregate labels from a crowd, we often use models like Dawid-Skene to estimate worker reliability. As Ofer Dekel and Ohad Shamir point out in Vox Populi: Collecting High-Quality Labels from a Crowd, these models fail to account for "temporal drift." If a worker's accuracy drops from 95% to 60% due to boredom, a static model will average these out, potentially trusting the worker too much during their "tired" phase and too little during their "expert" phase.
This non-stationarity manifests in several distinct ways:
- Learning Curves: In complex tasks like medical annotation or species identification, workers aren't experts from the start. Shengying Pan and Kate Larson demonstrate in Dynamic Task Allocation Algorithm for Hiring Workers that Learn that modeling worker improvement via hyperbolic learning curves can reduce hiring costs by 62%.
- Cognitive Fatigue and Boredom: Repetitive tasks lead to "position bias" or "lazy" clicking. Qianqian Xu and colleagues in False Discovery Rate Control and Statistical Quality Assessment of Annotators in Crowdsourced Ranking show that annotators often stop paying attention and default to clicking one side of the screen.
- Memory and Context: The order of tasks matters. Carlos Castillo 0001 and Valerie L. Shalin argue in Modeling and Mitigating Human Annotation Errors that human reliability is a function of recent task history, influenced by memory decay and attention slips.
Tracking the Moving Target
To move beyond static models, researchers are borrowing tools from time-series analysis and computational neuroscience.
One powerful approach is Visual Knowledge Tracing (VKT). Neehar Kondapaneni and Pietro Perona use recurrent neural networks to model the "non-stationary state" of a learner in Visual Knowledge Tracing. Instead of assuming a fixed internal classification function, they predict how a human’s perception of a category (like a specific bird species) shifts as they see more examples.
Similarly, Himabindu Lakkaraju and Jure Leskovec introduced Confusions over Time (CoT) in Confusions over Time: An Interpretable Bayesian Model to Characterize Trends in Decision Making. This framework audits human decisions—such as those made by judges or doctors—by modeling how their "confusion matrices" (the types of errors they make) evolve over a career. This allows for the diagnosis of systemic biases that only emerge during specific periods, such as late-afternoon fatigue or early-career inexperience.
Why Modeling Dynamics Works
The intuition behind these dynamic models is that error is not just noise; it is a signal of the annotator's state.
If we know that a worker is in a "learning" state, we can provide them with easier tasks or more feedback. If we detect a "fatigue" state, we can trigger a break or discount their labels. In Crowdsourcing with the drift diffusion model of decision making, Shamal Lalvani uses a neuroscience-inspired model to show that human performance is a dynamic accumulation of evidence, where attention levels fluctuate over time.
This perspective is particularly critical in Human-AI Collaboration. In high-stakes deferral pipelines—where an AI decides whether to handle a case or hand it to a human—the system must know the human's current reliability. As Vijay Keswani and Matthew Lease note in Designing Closed Human-in-the-loop Deferral Pipelines, if the human expert is experiencing fatigue, the AI should be less likely to defer to them, even if that human is normally highly accurate.
Beyond the Average
By embracing the "debatable" and shifting nature of human judgment, we create more robust AI. Frameworks like the one proposed by Beatriz Gomez Anson in Let's Agree to Disagree or the multitask learning approach by Uthman Jinadu in Noise Correction on Subjective Datasets suggest that we shouldn't just "clean" data to find a single truth. Instead, we should model the reasons for the noise—whether it's a temporal shift in an annotator's criteria or a genuine subjective disagreement.
The future of data collection and human-AI interaction lies in moving away from the "snapshot" view of human ability and toward a "cinematic" view that tracks the continuous, non-stationary evolution of human expertise.