Group Relative Policy Optimization for Text-to-Speech with Large Language Models

Chang Liu, Yajun Hu, Yingying Gao, Shilei Zhang, Zhen-Hua LingarXiv 2025 · DOI · arXiv

TL;DR

Researchers have successfully applied Group Relative Policy Optimization (GRPO)—the reinforcement learning algorithm behind DeepSeek-V3—to fine-tune Large Language Model (LLM)-based Text-to-Speech (TTS) systems. By using an off-the-shelf ASR model to provide a dual reward based on transcription accuracy and probabilistic confidence, the method significantly reduces pronunciation errors and improves speech naturalness without the need for expensive human preference data or complex "critic" models.

What Problem Does This Paper Address?

While LLM-based TTS models (like CosyVoice or Llasa) produce remarkably human-like speech, they suffer from a common "hallucination" problem: they occasionally skip words, repeat phrases, or mispronounce terms.

Standard training uses Maximum Likelihood Estimation (MLE), which teaches the model to predict the next token but doesn't explicitly optimize for how a human perceives the final audio. Previous attempts to fix this using Reinforcement Learning (RL) have been cumbersome. PPO (Proximal Policy Optimization) requires a separate "value model" (a critic), doubling the memory overhead. DPO (Direct Preference Optimization) avoids the extra model but requires massive amounts of high-quality paired data (e.g., "this audio is better than that audio"), which is expensive to create for speech.

Methods

The authors introduce a training pipeline that leverages GRPO to fine-tune TTS models.

1. The GRPO Advantage

Unlike PPO, GRPO eliminates the need for a value function. Instead, for every input text, the model generates a "group" of multiple speech outputs. It then calculates the reward for each and computes an "advantage" score based on how much better or worse an output is compared to the group average. This stabilizes training while being much more computationally efficient.

2. The Composite Reward Function

The "brain" of this approach is how it calculates rewards using an off-the-shelf ASR model (Whisper-large-v3). They combine two metrics:

  • Character Error Rate (CER): An intuitive measure of whether the model said the right words. However, CER is "discrete" (it doesn't change if the model is slightly more confident but still gets the word right).
  • Negative Log-Likelihood (NLL): This measures the ASR model's confidence in the transcript. It provides a continuous, fine-grained signal that helps the model optimize for subtle nuances like prosody and clarity.

The final reward is a weighted harmonic mean of these two. Using a harmonic mean ensures that if the model fails significantly in one area (e.g., a very high CER), the total reward drops sharply, preventing the model from "gaming" one metric while failing the other.

Datasets

  • Fine-tuning: 4,000 sentences sampled from the Emilia dataset, covering Chinese, English, Japanese, and Korean.
  • Objective Evaluation:
    • seed-tts-eval for Chinese (2,020 samples) and English (1,088 samples).
  • Common Voice for Japanese and Korean (1,000 samples each).
  • Subjective Evaluation: A subset of 100 highly expressive samples per language for listening tests.

Key Insights and Ideas

  • ASR as a "Zero-Cost" Teacher: By using a pre-trained ASR model to provide rewards, the researchers bypassed the need for human-labeled preference pairs. The ASR model acts as an objective judge of intelligibility.
  • The Power of NLL: The authors found that CER alone is often too "blunt." NLL provides the "gradient" needed for the model to understand how to improve even when the words are technically correct.
  • Stability through Grouping: Group-relative advantages prevent the reward signals from swinging wildly, which is a common failure mode when fine-tuning sensitive LLMs.

Key Results

The researchers tested their method on two architectures: Llasa-1B (acoustic token-based) and CosyVoice2 (semantic token-based).

  • Intelligibility Boost: For Llasa-1B in Chinese, the CER dropped from 7.73% to 1.30%—nearly reaching human-level performance (1.33%).
  • Naturalness (MOS): In subjective Mean Opinion Score (MOS) tests, the GRPO-tuned models consistently outperformed the baselines. For English CosyVoice2, the score rose from 4.22 to 4.43.
  • Error Correction: Qualitative analysis (spectrograms) showed that the fine-tuned models successfully corrected "omission" errors where the baseline model simply skipped difficult words.

Limitations and Future Work

The paper notes a specific limitation: while the method significantly improved the accuracy of Llasa-1B, it did not yield the same level of improvement in naturalness for that specific model compared to the semantic-token-based CosyVoice2. The authors plan to investigate the underlying reasons for this discrepancy in future work. Additionally, while the ASR model is a great judge of what was said, it may not be the perfect judge of emotional quality, suggesting a need for even more sophisticated reward functions in the future.

Summary updated

Abstract

This paper proposes a GRPO-based approach to enhance the performance of large language model (LLM)-based text-to-speech (TTS) models by deriving rewards from an off-the-shelf automatic speech recognition (ASR) model. Compared to previous reinforcement learning methods for LLM-based TTS, our method requires no dedicated model for reward computation or training. Moreover, we design a composite reward function that combines character error rate (CER) with negative log-likelihood (NLL) obtained from the ASR model, providing more informative and accurate reward signals. We apply GRPO fine-tuning to pre-trained LLM-based TTS models and evaluate their zero-shot TTS performance. Experimental results show that the proposed method substantially improves both the intelligibility and naturalness of synthesized speech. Ablation studies and further analyses confirm the effectiveness of integrating the two reward components.
BibTeX
@misc{liu2025group,
  author = {Chang Liu and Yajun Hu and Yingying Gao and Shilei Zhang and Zhen-Hua Ling},
  title = {{Group Relative Policy Optimization for Text-to-Speech with Large Language Models}},
  year = {2025},
  eprint = {2509.18798},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.18798},
  howpublished = {arXiv},
}
Figure