EyetrackingMOS: Proposal for an online evaluation method for speech synthesis models

Gustavo Araújo, Julio Cesar Galdino, Rodrigo Lima, Leonardo Ishida, Gustavo Lopes, Miguel Oliveira, Arnaldo Cândido, Sandra Maria Aluísio, Moacir Antonelli Ponti

Abstract

Evaluating Text-To-Speech (TTS) systems is challenging, as the increasing quality of synthesis makes it difficult to discriminate models’ ability to reproduce prosodic attributes, especially for Brazilian Portuguese. Offline evaluation metrics do not capture our genuine reactions to audio stimuli. Therefore, we propose an online evaluation method using eye-tracking. Our experiments with 76 annotators show a reasonable correlation between EyetrackingMOS and MOS, as well as a reduction in the total evaluation time. We believe this metric provides precise and potentially fast information to complement existing evaluation methods.

Introduction

As Text-to-Speech (TTS) systems approach human-level quality, particularly for languages like Brazilian Portuguese, traditional evaluation methods face significant limitations. The industry standard, Mean Opinion Score (MOS), is an "offline" metric where listeners assign a subjective rating (1-5) after hearing an audio clip. This approach is susceptible to cognitive bias, as listeners have time to deliberate, potentially filtering out immediate, instinctive reactions to prosodic errors such as rhythm, intonation, and stress. Furthermore, as synthesis quality improves, scores tend to cluster at the high end of the scale, reducing the granularity needed to distinguish between high-performing models.

To address these issues, we propose EyetrackingMOS, a novel "online" evaluation framework. By leveraging real-time gaze tracking, this method captures instinctive human reactions to synthesized speech, mapping gaze duration to a traditional 1-5 scale.

Methodology

The EyetrackingMOS framework utilizes a webcam-based eye-tracker (Webgazer.js via the Gorilla.sc platform). During the evaluation, participants listen to audio samples while viewing two visual anchors—for example, a human icon and a robot icon. The system records the percentage of time spent looking at each icon.

Comparison of the experimental procedures for EyetrackingMOS and traditional MOS.

As shown in Figure 1, the EyetrackingMOS procedure includes additional calibration steps compared to the traditional MOS flow. The gaze percentage is then mapped to the 1-5 MOS scale. For instance, if a user spends 81-100% of the time looking at the human icon, the clip is assigned a score of 5 (Excellent). The study utilized the MuPe-v1 dataset (spontaneous conversational speech) and the SYNTACC multi-accent TTS model for validation.

Results and Discussion

The experimental validation involved 76 annotators (28 experts and 48 non-experts). A key finding was the high consistency of gaze patterns across participants compared to the high variability often seen in traditional MOS ratings. This suggests that eye movements are more "mechanical" and synchronized with the stimulus, bypassing the deliberation that can lead to "nice" but inaccurate ratings.

Scatter plot showing the correlation between traditional MOS and EyetrackingMOS, categorized by human and synthetic speech samples.

Figure 3 illustrates the relationship between the two metrics. The results demonstrate a strong Pearson correlation of 0.744 and an $R^2$ value of 0.553, indicating that eye-tracking data is a reliable proxy for subjective human judgment. Crucially, the method effectively distinguishes between human and synthetic samples, as seen in the distinct clustering of data points.

EyetrackingMOS also offers improved granularity for high-quality audio. While MOS often hits a "ceiling" where most samples receive a 5, the continuous 0-100% gaze scale reveals subtle differences in how convincing a clip is to the listener. While total evaluation time was slightly longer due to calibration, the actual evaluation phase is potentially faster as it requires no manual input after each clip.

Conclusion and Future Work

EyetrackingMOS presents a promising complement to traditional TTS evaluation metrics. While webcam-based tracking introduces challenges related to environmental factors and calibration overhead, the method's ability to capture instinctive, real-time reactions provides deeper insights into speech naturalness.

Future research will focus on refining the interpretation of intermediate gaze positions and testing the robustness of the method in uncontrolled, crowdsourced environments. By reducing cognitive bias and increasing granularity, EyetrackingMOS moves us closer to a more precise understanding of human perception in speech synthesis.

Summary updated

Abstract

Evaluating Text-To-Speech (TTS) systems is challenging, as the increasing quality of synthesis makes it difficult to discriminate models’ ability to reproduce prosodic attributes, especially for Brazilian Portuguese. Offline evaluation metrics do not capture our genuine reactions to audio stimuli. Therefore, we propose an online evaluation method using eye-tracking. Our experiments with 76 annotators show a reasonable correlation between EyetrackingMOS and MOS, as well as a reduction in the total evaluation time. We believe this metric provides precise and potentially fast information to complement existing evaluation methods.
BibTeX
@inproceedings{araujo2024eyetrackingmos,
  author = {Gustavo Araújo and Julio Cesar Galdino and Rodrigo Lima and Leonardo Ishida and Gustavo Lopes and Miguel Oliveira and Arnaldo Cândido and Sandra Maria Aluísio and Moacir Antonelli Ponti},
  title = {{EyetrackingMOS: Proposal for an online evaluation method for speech synthesis models}},
  booktitle = {Stil},
  year = {2024},
  doi = {10.5753/stil.2024.245424},
}
Figure