HEMVIP: Human Evaluation of Multiple Videos in Parallel

Patrik Jonell, Youngwoo Yoon, Pieter Wolfert, Taras Kucherenko, Gustav Eje Henter

Abstract

In many research areas, for example motion and gesture generation, objective measures alone do not provide an accurate impression of key stimulus traits such as perceived quality or appropriateness. The gold standard is instead to evaluate these aspects through user studies, especially subjective evaluations of video stimuli. Common evaluation paradigms either present individual stimuli to be scored on Likert-type scales, or ask users to compare and rate videos in a pairwise fashion. However, the time and resources required for such evaluations scale poorly as the number of conditions to be compared increases. Building on standards used for evaluating the quality of multimedia codecs, this paper instead introduces a framework for granular rating of multiple comparable videos in parallel. This methodology essentially analyses all condition pairs at once. Our contributions are 1) a proposed framework, called HEMVIP, for parallel and granular evaluation of multiple video stimuli and 2) a validation study confirming that results obtained using the tool are in close agreement with results of prior studies using conventional multiple pairwise comparisons.

Introduction: The Evaluation Bottleneck

In fields like motion synthesis and gesture generation, objective metrics (like joint velocity or distance) rarely capture the "feel" or "appropriateness" of a movement. Consequently, human subjective evaluation remains the gold standard. However, current methods face significant scaling challenges. Likert scales (rating one video at a time) suffer from "standard drift," where raters struggle to maintain a consistent internal benchmark over time. Pairwise comparisons (choosing between Video A and Video B) are effective for spotting subtle differences but explode in complexity as the number of models increases; comparing 8 different systems requires 28 unique pairs.

HEMVIP (Human Evaluation of Multiple Videos in Parallel) addresses this bottleneck by providing a tool for granular, side-by-side comparison of many conditions simultaneously.

The HEMVIP Framework

HEMVIP is heavily inspired by the MUSHRA (MUltiple Stimuli with Hidden Reference and Anchor) standard used in audio codec testing. The framework allows researchers to evaluate multiple video stimuli—such as AI-generated gestures or animations—simultaneously on a single screen.

The HEMVIP user interface allows for parallel evaluation of multiple video stimuli using color-coded sliders.

Key features of the framework include:

  • Parallel Interface: Users interact with multiple "Play" buttons and corresponding sliders (0–100 scale) on a single page.
  • Synchronized Context: All videos on a page represent the same underlying data (e.g., the same speech segment) but are generated by different models, making differences easier to identify.
  • Visual Mapping: To reduce cognitive load, the interface uses color-coding—the border of the currently playing video matches the color of its corresponding rating slider.
  • Blind & Randomized: System labels are hidden, and the order of stimuli is randomized per page to prevent bias.
  • Attention Checks: The system includes "trap" stimuli where a voiceover or text overlay instructs the user to set a specific slider value, filtering out inattentive participants.
  • Technical Stack: The system is built using a web-based frontend (modified from webMUSHRA), a FastAPI backend, and a MongoDB database.

Validation Study

To validate HEMVIP, the authors reproduced a previous study on speech-driven gesture generation. They used a dataset of 10-second clips featuring a "Full" state-of-the-art model, human ground truth, and six ablation models (e.g., NoAudio, NoText, NoPCA). The study involved 184 participants recruited via the Prolific crowdsourcing platform.

Key Results and Discussion

The validation study demonstrated that HEMVIP is both efficient and statistically sensitive.

Comparison of significance results between HEMVIP and a prior pairwise study across various system contrasts.
  • High Correspondence: HEMVIP results agreed with previous pairwise studies in 23 out of 28 system contrasts. In cases where they differed, the direction of preference remained consistent, but HEMVIP proved more conservative or sensitive.
  • Statistical Sensitivity: The granular 0–100 scale allows for more sophisticated statistical tests (like t-tests or Wilcoxon signed-rank tests) that can identify significant differences with fewer samples compared to binary preference tests.
  • Scaling Efficiency: Comparing one system against seven others takes approximately 176 seconds with HEMVIP, compared to 224 seconds for pairwise tests. For all-to-all comparisons, the efficiency gain is even more dramatic (176s vs. ~900s).

Conclusion

HEMVIP provides a scalable, efficient, and statistically sensitive alternative to traditional subjective evaluation methods. While the authors note a cognitive load limit of approximately 12 videos per page, the framework successfully ranks model components and identifies subtle quality differences. Future work will explore multi-question interfaces and generalization to other video tasks like facial animation.

The code for HEMVIP is open-source and available on GitHub.

Summary updated

Abstract

In many research areas, for example motion and gesture generation, objective measures alone do not provide an accurate impression of key stimulus traits such as perceived quality or appropriateness. The gold standard is instead to evaluate these aspects through user studies, especially subjective evaluations of video stimuli. Common evaluation paradigms either present individual stimuli to be scored on Likert-type scales, or ask users to compare and rate videos in a pairwise fashion. However, the time and resources required for such evaluations scale poorly as the number of conditions to be compared increases. Building on standards used for evaluating the quality of multimedia codecs, this paper instead introduces a framework for granular rating of multiple comparable videos in parallel. This methodology essentially analyses all condition pairs at once. Our contributions are 1) a proposed framework, called HEMVIP, for parallel and granular evaluation of multiple video stimuli and 2) a validation study confirming that results obtained using the tool are in close agreement with results of prior studies using conventional multiple pairwise comparisons.
BibTeX
@inproceedings{jonell2021hemvip,
  author = {Patrik Jonell and Youngwoo Yoon and Pieter Wolfert and Taras Kucherenko and Gustav Eje Henter},
  title = {{HEMVIP: Human Evaluation of Multiple Videos in Parallel}},
  booktitle = {ICMI},
  year = {2021},
  url = {https://openreview.net/forum?id=U9O3Ci2ZVR},
}

Loading related content...

Figure