RedPen: Region- and Reason-Annotated Dataset of Unnatural Speech

Kyumin Park, Keon Lee, Daeyoung Kim, Dongyeop KangarXiv 2022 · OpenReview

Abstract

Recent advances in speech synthesis have significantly improved audio quality, yet evaluation remains largely dependent on the Mean Opinion Score (MOS). While MOS provides a single naturalness rating, it fails to explain which parts of the speech are unnatural or why. We introduce RedPen, a novel dataset featuring human annotations of unnatural speech regions and their corresponding error types. Comprising 180 synthesized samples, RedPen categorizes errors into 11 types, such as voice trembling and background noise. Our analysis demonstrates that human-annotated regions provide superior explanations compared to model-driven predictions and reveal distinct "error fingerprints" for different synthesis systems. This dataset aims to foster the development of more interpretable speech evaluation models.

The Problem: The "Black Box" of Naturalness Scores

In speech synthesis research, the gold standard for evaluation is the Mean Opinion Score (MOS), where human judges rate audio on a scale of 1 to 5. However, MOS is a "flat" metric. It indicates that a sample sounds unnatural but offers no diagnostic information—it cannot distinguish between a glitchy ending, an erratic pitch shift, or persistent background hiss.

Example of RedPen annotation showing specific regions of unnatural speech and their associated reasons.

Existing automated methods attempt to provide interpretability using saliency maps (e.g., Integrated Gradients) to guess which audio segments a model finds important. Unfortunately, these machine-generated intuitions often fail to align with actual human perception. RedPen addresses this gap by capturing the nuance of human "ear-witness" testimony, mapping specific errors to precise time intervals.

Methodology and Dataset Construction

The researchers developed a custom annotation pipeline to collect data in two distinct stages:

  1. Region Annotation: Crowd workers on Amazon Mechanical Turk identified continuous spans where speech sounded unnatural. To account for the inherent "fuzziness" of human timing, the team employed a binning-based approach (0.1s intervals) to calculate agreement.
  2. Reason Annotation: Because identifying the technical cause of an error requires expertise, speech experts categorized the highlighted regions into 11 error types: End of Speech (EOS), Silence, High Pitch, Voice Trembling, Flat Pitch, Energy-related, Spacing, Word duration, Accuracy (phoneme errors), Background noise, and Undefined.

Dataset Statistics

The resulting RedPen dataset consists of 180 synthesized speech samples from the Voice Conversion Challenge 2018 (VCC2018). Each sample includes annotations from three different participants. Validation showed an inter-annotator agreement (Cohen’s Kappa) of 0.697, indicating high consistency in how humans identify unnatural audio segments.

Key Insights: Model Fingerprints and the Interpretation Gap

The Interpretation Gap

A critical finding of this research is that model-driven saliency (Integrated Gradients) has almost zero correlation (approximately 0.015) with human-annotated error regions. This suggests that current "interpretable" Text-to-Speech (TTS) models are not actually focusing on the features that human listeners find jarring.

Model Fingerprints

RedPen reveals that different TTS systems possess unique "error profiles." For example, one system might struggle primarily with voice trembling, while another is prone to phoneme accuracy errors. By visualizing these occurrences, developers can pinpoint specific architectural weaknesses rather than relying on a generic low score.

Comparison of error type occurrences across different synthesis systems, highlighting unique model fingerprints.

Quantifying Quality

The data confirms a strong negative correlation (-0.687) between the total length of unnatural regions and the overall naturalness score. Essentially, the longer the "cringe" moment lasts, the lower the MOS—a relationship that is now formally quantified through RedPen.

User Preferences in Evaluation

In preference tests, users significantly favored human-annotated regions over model-predicted ones (57.5% vs 42.5%) when asked which better represented unnaturalness. Furthermore, when presented with different evaluation interfaces, users preferred a "Score + Region" display (41.3%) over a simple score (26.7%). Interestingly, adding the specific "Reason" (Score + Region + Reason) was slightly less preferred (32.0%), suggesting that while the data is valuable for researchers, it may increase interface complexity for general users.

Future Directions

While RedPen provides high-quality "boutique" data, the authors acknowledge the need to scale the dataset to thousands of samples to train robust automatic error-detection models. Currently, the "reason" marking stage requires speech experts, creating a bottleneck. Future work will focus on simplifying these categories for general crowd workers to accelerate data collection. The ultimate goal is to move toward an automated "Red Pen" for speech synthesis—evaluation models that don't just output a 1–5 score, but actively highlight and label errors for researchers.

Summary updated

Abstract

Even with recent advances in speech synthesis models, the evaluation of such models is based purely on human judgement as a single naturalness score, such as the Mean Opinion Score (MOS). The score-based metric does not give any further information about which parts of speech are unnatural or why human judges believe they are unnatural. We present a novel speech dataset, RedPen, with human annotations on unnatural speech regions and their corresponding reasons. RedPen consists of 180 synthesized speeches with unnatural regions annotated by crowd workers; These regions are then reasoned and categorized by error types, such as voice trembling and background noise. We find that our dataset shows a better explanation for unnatural speech regions than the model-driven unnaturalness prediction. Our analysis also shows that each model includes different types of error types. Summing up, our dataset successfully shows the possibility that various error regions and types lie under the single naturalness score. We believe that our dataset will shed light on the evaluation and development of more interpretable speech models in the future. Our dataset will be publicly available upon acceptance.
BibTeX
@misc{park2022redpen,
  author = {Kyumin Park and Keon Lee and Daeyoung Kim and Dongyeop Kang},
  title = {{RedPen: Region- and Reason-Annotated Dataset of Unnatural Speech}},
  year = {2022},
  eprint = {2210.14406},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.14406},
}
Figure