Machine learning models are only as good as the data used to train them, and building high-quality datasets relies fundamentally on human annotators. To handle the inevitable mistakes made by human labelers, standard quality control frameworks—ranging from classic majority voting to complex probabilistic models like Dawid-Skene—rely on a foundational assumption: an annotator's expertise is a static trait. They treat a labeler's reliability, bias, and error rates as fixed parameters throughout an annotation session.
However, humans are not static, label-generating oracles. In reality, human performance is highly non-stationary. As an annotator progresses through a task, their internal criteria, attention span, and accuracy drift. As Dekel et al. and Khattak point out, ignoring this temporal drift fails to account for critical human factors like fatigue, boredom, or the gradual acquisition of task-specific expertise. By treating every label as an independent event, traditional models fail to mitigate the structured, time-varying noise introduced by shifting human judgment (Gomez Anson et al.).
To build more robust machine learning systems, a growing body of research focuses on uncovering why human performance fluctuates over time and how algorithms can dynamically model these temporal trajectories.
The Dual Forces of Temporal Drift: Fatigue and Learning
Temporal variance in human labeling is not completely random; it typically follows predictable behavioral patterns driven by two opposing forces: performance degradation (fatigue) and performance enhancement (learning).
Fatigue and Cognitive Load
During extended labeling sessions, annotators experience cognitive exhaustion, stress, or boredom. This physical and psychological strain causes their internal labeling criteria to undergo a temporal distribution shift (Jinadu et al.). Kiyasseh demonstrates that even highly trained medical experts suffer from fluctuating error rates due to fatigue, leading to highly inconsistent label quality across a single session.
Learning and Task Familiarity
Conversely, annotators often refine their understanding of a task as they encounter more examples. This means their internal classification logic and clustering criteria actively evolve over time (Gomes et al.). For instance, Bukas et al. observed that as annotators gain experience within a dataset, they often become significantly more selective, ignoring ambiguous or smaller structures that they might have labeled initially.
Even subtle psychological biases change over time. Fomicheva et al. discovered that human susceptibility to "reference bias" varies across a session: judges are most capable of recognizing valid semantic variations when they are fully warmed up, but this capability degrades rapidly at the beginning and end of a session due to initial unfamiliarity and eventual cognitive fatigue.
[ Session Start ] ──► [ Warmed Up ] ────────► [ Extended Session ]
Unfamiliarity Peak Expertise Cognitive Fatigue
High Bias High Consistency Shifting Criteria
From Independent Labels to Sequential Trajectories
If an annotator's reliability is a moving target, how can an algorithm accurately estimate the ground truth? The core mechanism behind modern dynamic worker modeling is a shift from treating labels as isolated data points to treating them as a continuous sequential trajectory.
Instead of assigning a single static accuracy score to a worker, newer frameworks track behavioral history. For example, Yang et al. introduced a temporal-aware framework (TACMA) that models a worker's short-term memory, dynamically weighting their current labels based on their most recent sequence of annotations. This prevents deep neural networks from memorizing the non-stationary noise produced when a worker loses focus.
Other approaches explicitly treat the human annotator as an active learner within the loop. Tang proposed a bidirectional active learning framework (BALT) that strategically intersperses "gold instances" (samples with known ground truth) throughout a session. By evaluating how an annotator performs on these anchor points over time, the system can dynamically map their learning curve or fatigue rate, allowing it to adaptively decide whether to trust their latest labels, retrain the worker, or retroactively correct their previous mistakes.
By replacing static reliability parameters with dynamic, time-dependent functions, aggregation models can weight an individual's votes based precisely on when they were cast during the session (Audhkhasi; Liu et al.).
Systemic Consequences Across the ML Lifecycle
Failing to model human non-stationarity doesn't just result in slightly noisier datasets; it introduces systemic biases that ripple across dataset replication, active learning, and human-AI collaboration.
- Dataset Replication Crises: Engstrom et al. demonstrated that the massive accuracy drops observed when testing models on replicated benchmarks (like ImageNet-v2) are largely driven by subtle statistical biases introduced by "task shift." Because the subjective criteria humans use to categorize data inevitably evolve over time, perfectly replicating a historical benchmark becomes impossible if the underlying human interpretation of the task has drifted.
- Suboptimal Active Learning: Standard active learning algorithms select the most informative data points assuming an oracle will label them accurately. However, Palmer notes that because human expertise evolves during data collection, the "gold standard" itself is a moving target. Selecting highly complex examples can backfire, dramatically increasing cognitive load and accelerating worker fatigue if the algorithm does not match the difficulty of the sample to the annotator's current position on their learning curve.
- Brittle Human-AI Teams: In deployment, systems designed to defer difficult tasks to human experts typically treat human reliability as a static variable (Wei et al.). Yet human trust, vigilance, and performance are highly dynamic. As Yu demonstrates, human trust in an AI teammate fluctuates based on the machine's real-time accuracy, meaning that human-AI collaboration frameworks must model the non-stationary feedback loops between both agents to remain safe and effective.