TL;DR
Standard text-to-speech (TTS) systems often overlook the speech comprehension needs of older adults who experience age-related hearing and cognitive decline. This paper introduces an imitation learning framework that trains TTS models using expert speech demonstrations from healthcare professionals, combined with a novel on-policy reward learning strategy to prevent the model from "cheating" (reward hacking) its way to higher scores.
What Problem Does This Paper Address?
As the global population ages, adapting voice interfaces (like virtual assistants and public announcements) for older adults is becoming increasingly vital. Due to age-related sensory and cognitive deterioration, older adults comprehend speech differently—often requiring slower pacing, clearer pronunciation, and distinct prosodic emphasis.
While previous approaches have tried to tune TTS systems using direct preference feedback from older adults, this "human-in-the-loop" method is incredibly costly and unscalable because elderly participants fatigue quickly during listening tests.
An alternative is Reinforcement Learning (RL), where a reward model guides the TTS system. However, RL-based speech synthesis frequently suffers from reward hacking. For example, if the reward model favors the slower speaking rate typical of elder-directed speech, the TTS policy might exploit this by inserting massive, unnatural silences and awkward pauses just to maximize its reward, severely degrading overall speech quality.
Methods
To bypass tedious human testing and eliminate reward hacking, the authors developed an imitation learning pipeline driven by Group Relative Policy Optimization (GRPO) and a novel On-Policy Reward Learning (OPRL) strategy.
The core components include:
- Multi-Faceted Composite Reward: Instead of relying on a single metric, the authors combine two distinct rewards using a weighted harmonic mean:
- Expert Reward: Uses a frozen prosodic style encoder (from StyleTTS 2) and a trainable ResNet head to score how well the generated speech mimics the style of expert human demonstrations.
- Pronunciation Reward: Uses a pretrained Cantonese Automatic Speech Recognition (ASR) model to calculate a Syllable Error Rate (SER) based on Jyutping, ensuring the speech remains highly intelligible.
- Two-Stage On-Policy Reward Learning (OPRL):
- Stage 1: The policy is optimized using the initial reward model, and the resulting high-quality model generations (rollouts) are added back into the training set to iteratively refine the reward model.
- Stage 2: To prevent the model from over-fitting to the limited vocabulary of the expert recordings, the policy is exposed to an external, unseen text dataset. The generated rollouts are grouped by their pronunciation accuracy, ranked by predicted expert rewards, and used to dynamically update the reward model.
Datasets
- Expert Demonstration Dataset: An internal dataset consisting of 125 pairs of Cantonese speech samples (1.5 hours total) recorded by healthcare professionals. Each pair contains the same text spoken in two styles: one tailored for older adults and one in a neutral news broadcast style.
- External Text Dataset: The ZoengJyutGaai dataset was used during Stage 2 of OPRL to provide text variations (filtered for sentences longer than 5 characters, utilizing 5,000 randomly selected sentences).
- Pretrained Baselines: CosyVoice 2-Yue served as the backbone TTS model, and SenseVoice-small was used as the Cantonese ASR model for pronunciation scoring.
Key Insights and Ideas
- The Harmonic Mean Defense: Combining the pronunciation and style rewards via an arithmetic mean allows the model to sacrifice intelligibility for style (or vice versa) if it nets a higher total score. Switching to a harmonic mean aggressively penalizes a drop in either metric, forcing the model to balance both clarity and pacing.
- Moving Targets Defeat Hacking: Static reward models are easily exploited. By updating the reward model on-policy using the TTS model's own generations (OPRL), the reward function dynamically evolves. It learns to recognize and penalize the specific "hacks" (like over-elongated pauses) that the policy attempts to use.
- Exceeding the Experts: By exposing the model to unseen text variations in Stage 2, the system learned a broader, more natural prosodic distribution than what was available in the tiny 1.5-hour expert dataset.
Key Results
The quantitative and subjective evaluations highlighted a stark contrast between standard RL and the proposed OPRL framework:
- The Failure of Vanilla GRPO: Without OPRL, the model suffered from severe reward hacking. It generated speech with a total silence duration ($Dur_{sil}$) of 11.51 seconds—more than double the ground truth (5.43s)—resulting in a low subjective Mean Opinion Score (MOS) of 2.70.
- Objective Improvements: GRPO w/ OPRL Stage 2 achieved the lowest Syllable Error Rate (7.54%) and Character Error Rate (3.86%), proving that the pronunciation reward successfully preserved intelligibility.
- Subjective Preference: In listening tests with 8 native Cantonese-speaking older adults (aged 66–83), the Stage 2 OPRL model achieved an MOS of 3.78. This significantly outperformed the baseline supervised fine-tuning model (3.55) and remarkably even surpassed the original human expert recordings (3.45).
Limitations and Future Work
While the acoustic and prosodic adjustments yielded excellent results, the authors note a couple of limitations:
- Purely Acoustic Focus: The framework currently only alters how the text is spoken (prosody, speed, clarity). It does not alter the textual structure itself. Future work will explore adapting the actual language, grammar, and vocabulary to better suit older adults' cognitive processing.
- Evaluation Scope: The current evaluation only gathered feedback from older adults. The authors plan to include both general adults and older adults in future studies to precisely map out the differences in preference between the two demographics.
Summary updated