TL;DR
This paper investigates whether non-native English speakers (NNSs) judge the quality of synthesized speech similarly to native English speakers (NSs). While both groups generally agree on which synthesized speech samples are better, NNSs exhibit higher variance in their judgments, suggesting that their perception of synthetic speech quality is more idiosyncratic. This has implications for how TTS systems are evaluated and deployed for diverse user bases.
What Problem Does This Paper Address?
Text-to-Speech (TTS) systems are becoming ubiquitous, but their evaluation often relies solely on native speaker (NS) judgments. This overlooks the reality that many users of TTS systems, especially in widely spoken languages like English, are non-native speakers (NNSs). Previous research has shown that NNSs can have different perceptual experiences, particularly with intelligibility tasks when noise is present. However, it's unclear if these differences extend to subjective quality judgments of synthesized speech, and if NNSs' perceptions are consistent enough to be reliably studied. This paper aims to bridge this gap by directly comparing NS and NNS judgments of synthetic speech quality.
Methods
The researchers employed two standard methods for evaluating speech quality:
- Multidimensional Scaling (MDS): This technique allows for a visual representation of how stimuli are perceived relative to each other. Participants were asked to make pairwise comparisons, deciding if two speech samples were the same or different in naturalness. This method provides a rich, multi-dimensional view of perceptual space.
- Mean Opinion Score (MOS): This is a more direct rating method where participants score stimuli on a scale (in this case, 1-10) for naturalness.
The study used a combination of these methods to see if the richer output from MDS would align with MOS results and to compare the judgments of NSs and NNSs.
Datasets
The study used speech stimuli from the Blizzard Challenge 2008 test set A. This dataset is a well-established benchmark for TTS system evaluation. The stimuli included:
- Pairs of sentences from four high-ranking TTS systems.
- Two natural speech recordings to serve as anchors for perceived quality.
The specific texts of the spoken sentences are provided in Table 1 of the paper.
Key Insights and Ideas
- NNSs are a Diverse Group: The paper highlights that NNSs are not a monolithic group. Their perception of English is influenced by their individual language competence and their native language (L1), making generalizations difficult without specific subgroup analysis.
- General Agreement, but Higher Variance: Despite the diversity, NSs and NNSs generally agree on the overall quality ranking of synthesized speech samples. They can both distinguish natural speech from synthesized speech and tend to group similar quality stimuli together.
- NNSs' Judgments are More "Noisy": The key finding is that NNSs exhibit significantly higher variance in their judgments compared to NSs. This means their individual ratings are more spread out, making it harder to draw firm conclusions from smaller NNS participant groups.
- Potential for Transferability: While NNSs are more variable, the study suggests that if NSs find a synthesized speech sample to be of high quality, NNSs are likely to agree. This implies that NS evaluations might still be a useful proxy for NNS perception, at least for identifying generally good systems.
- Different Sensitivity to Errors: The paper hints that NNSs might be less sensitive to certain types of prosodic or intonation errors that strongly affect NS perception. This could explain some of the observed differences in ranking.
Key Results
- MDS Representations: MDS plots for both NSs and NNSs showed similar clustering patterns, with natural speech samples clearly separated from synthesized ones.
- Rank Agreement: Both groups generally agreed on the relative ranking of the synthesized speech stimuli, as confirmed by both MDS and MOS scores (Table 2).
- Stimulus T4 Anomaly: A notable difference was observed for stimulus T4, which NSs ranked in the middle-to-lower tier, while NNSs placed it much higher. This difference was also reflected in the MOS scores.
- Higher Variance in NNS Judgments: Statistical tests (Kolmogorov-Smirnov Z test) confirmed that NNS data introduced significantly more stress (a measure of fit) into the MDS representations, indicating greater variability in their judgments.
- MOS Tendencies: While generally agreeing, NSs tended to use the extreme ends of the MOS scale more frequently than NNSs.
Limitations and Future Work
- Generalizability of NNS Group: The paper acknowledges that the NNS group was self-selecting and diverse in terms of L1. Future work could focus on analyzing specific subgroups of NNSs (e.g., by L1) to understand finer-grained perceptual differences.
- "Noisy" Data: The higher variance in NNS judgments means that more participants would be needed to achieve statistically robust results, increasing the cost and time of evaluations.
- Transferability Hypothesis: While the data suggests positive results from NS evaluations can be transferred to NNSs, this is presented as a hypothesis. Further research is needed to confirm the extent and conditions under which this transferability holds.
- Specific Error Sensitivity: The paper speculates on NNSs' differing sensitivity to certain errors (e.g., prosody, intonation). Future work could involve targeted experiments to isolate and quantify the impact of specific acoustic and prosodic features on NNS perception.
Summary updated