PREFAB: PREFerence-based Affective Modeling for Low-Budget Self-Annotation

Jaeyoung Moon, Y.w. Choi, Yucheon Park, Dávid Melhárt, Georgios N. Yannakakis, Kyung-Joong Kim

TL;DR

PREFAB is a novel framework that slashes the effort required for "self-annotation"—the process where users label their own emotional states after a task. By using preference learning to identify key "inflection regions" (peaks and valleys of emotion) and interpolating the rest, it maintains high data quality while significantly reducing cognitive load and annotator fatigue.

What Problem Does This Paper Address?

In the field of Affective Computing, researchers need "ground truth" labels to know what a user is feeling. The gold standard is retrospective self-annotation: a user performs a task (like playing a video game), watches a recording of it, and uses a slider to report their levels of arousal or valence.

The problem? It’s exhausting. If a session lasts 20 minutes, the user has to spend another 20 minutes in a state of high concentration, constantly adjusting a slider. This leads to "decision fatigue," memory errors, and a decline in label quality over time. While some researchers have tried to use physiological sensors (like heart rate monitors) to find "important" moments to label, these sensors are intrusive and don't provide a way to fill in the gaps between those moments.

Methods

The authors introduce PREFAB (PREFerence-based Affective modeling), which moves away from "full annotation" toward a "low-budget" selective approach. Their method is built on two pillars:

  1. The Peak-End Rule: A psychological heuristic suggesting that our memories of an experience are defined by its most intense point (the peak) and its conclusion (the end).
  2. Ordinal Modeling: Instead of asking "How happy are you from 1 to 10?", PREFAB uses Preference Learning. It trains a model to look at two segments of a task and decide which one has higher emotional intensity.

The Technical Pipeline: * Siamese Transformer Network: The model uses a Transformer backbone to handle time-series data (game logs and video frames). It is conditioned using FiLM (Feature-wise Linear Modulation) to account for individual user biographies (age, gaming experience). * Inflection Detection: The model predicts a relative arousal curve and automatically identifies "inflection points"—the peaks and valleys where emotion changes direction. * Selective Annotation: Users are only asked to annotate 5-second clips centered around these inflection points. * Cumulative Slope Propagation: The system then uses a specialized linear interpolation algorithm to "stitch" these sparse labels together into a continuous emotional trace.

Datasets

The researchers evaluated PREFAB using two primary sources:

  • The AGAIN Dataset: A large-scale dataset containing play logs, gameplay videos, and arousal self-annotations from over 120 participants across nine different games. This was used for the initial technical validation.
  • Primary User Study: The authors conducted their own study with 25 participants playing TopDown (a shooter game). This study compared three conditions: full annotation (baseline), PREFAB without previews, and PREFAB with a "preview mechanism."

Key Insights and Ideas

  • Ordinal > Cardinal: One of the biggest "aha moments" is that modeling relative changes (is this more or less intense than before?) is far more robust than trying to predict absolute values. Absolute (cardinal) models are highly sensitive to noise and produce "jittery" curves that create too many false inflection points.
  • The "Preview" is the Secret Sauce: Simply showing a user a 5-second clip to label isn't enough; they lose the "vibe" of the session. The authors found that providing a brief contextual preview before the annotation window significantly boosted user confidence and label accuracy.
  • Interpolation is "Good Enough": The study proves that we don't actually need a human to label every single second. Linear interpolation between key emotional shifts captures the global trend of the experience well enough to match the quality of full-length manual traces.

Key Results

  • Technical Performance: PREFAB significantly outperformed random, uniform, and rule-based sampling methods. It achieved the best balance of F1 score (overlap with ground truth inflection points) and Time-Efficiency Alignment.
  • Reduced Workload: In the user study, PREFAB with previews significantly reduced both mental and physical demand compared to the baseline.
  • Increased Confidence: Interestingly, users felt more confident in their labels using PREFAB than they did during full annotation. The authors attribute this to the reduction in "monotonous" slider-dragging, which allowed users to focus only on the moments that actually mattered.
  • High Consistency: The reconstructed traces from PREFAB showed a high correlation with full-annotation ground truth (Concordance Correlation Coefficient = 0.67; Spearman’s $\rho$ = 0.69).

Limitations and Future Work

  • The Time Trade-off: While PREFAB reduces cognitive load, the "Time Efficiency" (TE) gains were conditional. If a session has many inflection points, watching the previews can sometimes take as long as (or longer than) the original session.
  • The "Cold Start" Problem: PREFAB requires initial annotated data to train the preference-learning model. It’s a "budget" method for future annotations, but you still need an upfront investment in data collection.
  • Simple Interpolation: The current system uses linear interpolation. The authors suggest that future versions could use more sophisticated, non-linear strategies to better mimic the "nuance" of human emotional decay or buildup.
  • Generalization: The user study was limited to one game genre. Future work is needed to see if these inflection-detection models transfer well across vastly different types of media, like film or educational content.

Summary updated

Abstract

Self-annotation is the gold standard for collecting affective state labels in affective computing. Existing methods typically rely on full annotation, requiring users to continuously label affective states across entire sessions. While this process yields fine-grained data, it is time-consuming, cognitively demanding, and prone to fatigue and errors. To address these issues, we present PREFAB, a low-budget retrospective self-annotation method that targets affective inflection regions rather than full annotation. Grounded in the peak-end rule and ordinal representations of emotion, PREFAB employs a preference-learning model to detect relative affective changes, directing annotators to label only selected segments while interpolating the remainder of the stimulus. We further introduce a preview mechanism that provides brief contextual cues to assist annotation. We evaluate PREFAB through a technical performance study and a 25-participant user study. Results show that PREFAB outperforms baselines in modeling affective inflections while mitigating workload (and conditionally mitigating temporal burden). Importantly PREFAB improves annotator confidence without degrading annotation quality.
BibTeX
@inproceedings{moon2026prefab,
  author = {Jaeyoung Moon and Y.W. Choi and Yucheon Park and Dávid Melhárt and Georgios N. Yannakakis and Kyung-Joong Kim},
  title = {{PREFAB: PREFerence-based Affective Modeling for Low-Budget Self-Annotation}},
  booktitle = {CHI},
  year = {2026},
  doi = {10.1145/3772318.3791702},
}
Figure