TL;DR
Human annotators are notoriously inconsistent when providing real-time, continuous ratings for subjective concepts like "movie violence" or "student engagement." This paper introduces a novel ground-truth generation pipeline that treats these noisy ratings as relative trends rather than absolute values, using supplemental "A vs. B" comparisons to anchor the data. The result is a massive leap in accuracy, boosting correlation with expert benchmarks from a dismal 0.15 to a near-perfect 0.95.
What Problem Does This Paper Address?
In fields like psychology and human-computer interaction, researchers often need "ground truth" labels for subjective states that change over time—think of a person’s stress levels during a task or the perceived intensity of a movie scene. The standard approach is to have several people provide continuous ratings (e.g., moving a slider from 0 to 1 while watching a video) and then average those ratings.
However, the authors identify three major "threats" that make simple averaging unreliable:
- Systemic Momentary Errors: Humans often "overshoot" when a change occurs or adjust their ratings even when the stimulus hasn't changed.
- Inconsistent Valuation: One person’s "0.8" intensity might be another person’s "0.5," and an individual's own internal scale can drift over the course of an hour.
- Feature Contamination: Some automated methods try to "fix" human labels by looking at the video/audio features, which risks creating a circular logic where the model only learns what the features tell it, not what the human actually perceived.
Methods
The authors propose a "paradigm shift" by treating continuous annotation as an ordinal (relative) process rather than an interval (absolute) one. Their pipeline consists of several sophisticated stages:
- SDA (Signed Differential Agreement): Instead of checking if annotators agree on the value (e.g., 0.7), this metric checks if they agree on the direction of change (is it going up, down, or staying flat?).
- Trapezoidal Segmented Regression (TSR): This algorithm simplifies noisy, jagged human rating lines into clean segments of "increasing," "decreasing," or "constant" intensity.
- Trend Voting: A majority-rule approach determines the consensus direction of the construct at every second.
- Ordinal Comparisons & Embedding: The pipeline identifies "flat" regions where the intensity is stable. It then asks crowd-workers to perform pairwise comparisons (e.g., "Which of these two clips is more violent?"). These comparisons are fed into a t-STE (t-stochastic triplet embedding) solver to map the clips onto a consistent 1D scale.
- Signal Deforming: Finally, the anchored "flat" segments are reconnected using the previously determined trends to create a smooth, continuous, and highly accurate ground-truth signal.
Datasets
The researchers introduced a new dataset to test their pipeline:
- Movie Violence Dataset: Continuous annotations of perceived violence in five Hollywood films (The Hustle, Good Boys, The Peanut Butter Falcon, The Possession of Hannah Grace, and Rambo: Last Blood).
- Annotators: 10 crowd-sourced workers per movie clip via Amazon Mechanical Turk.
- Benchmark: The results were validated against summarized violence ratings from Common Sense Media (CSM), which serves as a proxy for expert/authority consensus.
Key Insights and Ideas
- Trends > Values: The most reliable information a human provides in real-time is the change in a state, not the absolute magnitude. We are much better at noticing "it just got more violent" than deciding if the violence is exactly a "7.2 out of 10."
- The Power of Comparison: Pairwise comparisons (A vs. B) are cognitively easier and more stable than absolute ratings. By using a few thousand of these "anchors," the researchers could correct for the massive "scaling drift" found in continuous traces.
- Avoiding the "Average" Trap: Simple averaging fails because human errors in continuous annotation are structured (systemic), not random noise. Averaging "overshoots" doesn't cancel them out; it just creates a blurry, incorrect ground truth.
Key Results
The performance gains were dramatic. When comparing the generated ground truth to expert CSM ratings:
- Baseline (Simple Averaging): Achieved a Spearman correlation of only .15. This suggests that standard methods for fusing crowd-sourced continuous labels are surprisingly poor at capturing "expert" consensus.
- Proposed Pipeline: Achieved a Spearman correlation of .95.
- Qualitative Success: The pipeline successfully filtered out "low-effort" annotators and corrected for the tendency of crowd-workers to use the extreme ends of the rating scale inconsistently.
Limitations and Future Work
- Data Requirements: The pipeline requires a significant number of additional pairwise comparisons (over 20,000 in this study), which can be expensive and time-consuming to collect.
- Scale Anchoring: While the pipeline creates a perfect relative scale, the values are not "anchored" to a universal unit. Future work could involve using "anchor stimuli" with known values to make results comparable across different studies.
- Generalization: The study focused specifically on movie violence. The authors suggest the method should work for other constructs like "stress" or "engagement," but this remains to be empirically tested in those specific domains.
Reference: Booth, B. M., & Narayanan, S. S. (2024). People make mistakes: Obtaining accurate ground truth from continuous annotations of subjective constructs. Behavior Research Methods.
Summary updated