Abstract
Recent advances in speech synthesis have significantly improved audio quality, yet evaluation remains largely dependent on the Mean Opinion Score (MOS). While MOS provides a single naturalness rating, it fails to explain which parts of the speech are unnatural or why. We introduce RedPen, a novel dataset featuring human annotations of unnatural speech regions and their corresponding error types. Comprising 180 synthesized samples, RedPen categorizes errors into 11 types, such as voice trembling and background noise. Our analysis demonstrates that human-annotated regions provide superior explanations compared to model-driven predictions and reveal distinct "error fingerprints" for different synthesis systems. This dataset aims to foster the development of more interpretable speech evaluation models.
The Problem: The "Black Box" of Naturalness Scores
In speech synthesis research, the gold standard for evaluation is the Mean Opinion Score (MOS), where human judges rate audio on a scale of 1 to 5. However, MOS is a "flat" metric. It indicates that a sample sounds unnatural but offers no diagnostic information—it cannot distinguish between a glitchy ending, an erratic pitch shift, or persistent background hiss.
Existing automated methods attempt to provide interpretability using saliency maps (e.g., Integrated Gradients) to guess which audio segments a model finds important. Unfortunately, these machine-generated intuitions often fail to align with actual human perception. RedPen addresses this gap by capturing the nuance of human "ear-witness" testimony, mapping specific errors to precise time intervals.
Methodology and Dataset Construction
The researchers developed a custom annotation pipeline to collect data in two distinct stages:
- Region Annotation: Crowd workers on Amazon Mechanical Turk identified continuous spans where speech sounded unnatural. To account for the inherent "fuzziness" of human timing, the team employed a binning-based approach (0.1s intervals) to calculate agreement.
- Reason Annotation: Because identifying the technical cause of an error requires expertise, speech experts categorized the highlighted regions into 11 error types: End of Speech (EOS), Silence, High Pitch, Voice Trembling, Flat Pitch, Energy-related, Spacing, Word duration, Accuracy (phoneme errors), Background noise, and Undefined.
Dataset Statistics
The resulting RedPen dataset consists of 180 synthesized speech samples from the Voice Conversion Challenge 2018 (VCC2018). Each sample includes annotations from three different participants. Validation showed an inter-annotator agreement (Cohen’s Kappa) of 0.697, indicating high consistency in how humans identify unnatural audio segments.
Key Insights: Model Fingerprints and the Interpretation Gap
The Interpretation Gap
A critical finding of this research is that model-driven saliency (Integrated Gradients) has almost zero correlation (approximately 0.015) with human-annotated error regions. This suggests that current "interpretable" Text-to-Speech (TTS) models are not actually focusing on the features that human listeners find jarring.
Model Fingerprints
RedPen reveals that different TTS systems possess unique "error profiles." For example, one system might struggle primarily with voice trembling, while another is prone to phoneme accuracy errors. By visualizing these occurrences, developers can pinpoint specific architectural weaknesses rather than relying on a generic low score.
Quantifying Quality
The data confirms a strong negative correlation (-0.687) between the total length of unnatural regions and the overall naturalness score. Essentially, the longer the "cringe" moment lasts, the lower the MOS—a relationship that is now formally quantified through RedPen.
User Preferences in Evaluation
In preference tests, users significantly favored human-annotated regions over model-predicted ones (57.5% vs 42.5%) when asked which better represented unnaturalness. Furthermore, when presented with different evaluation interfaces, users preferred a "Score + Region" display (41.3%) over a simple score (26.7%). Interestingly, adding the specific "Reason" (Score + Region + Reason) was slightly less preferred (32.0%), suggesting that while the data is valuable for researchers, it may increase interface complexity for general users.
Future Directions
While RedPen provides high-quality "boutique" data, the authors acknowledge the need to scale the dataset to thousands of samples to train robust automatic error-detection models. Currently, the "reason" marking stage requires speech experts, creating a bottleneck. Future work will focus on simplifying these categories for general crowd workers to accelerate data collection. The ultimate goal is to move toward an automated "Red Pen" for speech synthesis—evaluation models that don't just output a 1–5 score, but actively highlight and label errors for researchers.
Summary updated