Abstract
The generalized command response (GCR) model represents intonation as a superposition of muscle responses to spike command signals. We have previously shown that the spikes can be predicted by a two-stage system, consisting of a recurrent neural network and a post-processing procedure, but the responses themselves were fixed dictionary atoms. We propose an end-to-end neural architecture that replaces the dictionary atoms with trainable second-order recurrent elements analogous to recursive filters. We demonstrate gradient stability under modest conditions, and show that the system can be trained by imposing temporal sparsity constraints. Subjective listening tests demonstrate that the system can synthesize intonation with high naturalness, comparable to state-of-the-art acoustic models, and retains the physiological plausibility of the GCR model.
Introduction
Intonation—the melodic rise and fall of pitch in speech—is a critical component for conveying emotion, emphasis, and linguistic structure. In modern Speech-to-Text (TTS) systems, achieving natural intonation is the primary hurdle to moving beyond robotic-sounding synthesis.
Traditionally, the Fujisaki Model and its successor, the Generalized Command Response (GCR) model, have described intonation as a physiological process. In this framework, "spiky" nerve commands trigger physical muscle responses, which in turn generate the fundamental frequency ($F_0$) of the voice. While physiologically grounded, previous neural implementations of the GCR model suffered from several limitations: the muscle response filters were fixed and not optimized during training, and the systems required a non-differentiable "post-processing" step that prevented the network from learning directly from pitch errors.
The End-to-End GCR Architecture
To address these flaws, we propose an End-to-End (E2E) GCR network. This architecture embeds the muscle models directly into the neural network as a specialized, trainable layer, allowing for full backpropagation from the final pitch output.
As shown in the figure above, the proposed system moves the boundary of the "trained" portion of the network. In the previous model, the RNN predicted parameters that were then handled by an untrained post-processing block. In our E2E model, the muscle models themselves are part of the trained parameters, allowing the system to discover the optimal "physics" for a specific speaker's voice.
Trainable Muscle Models
The muscle models are implemented as second-order linear all-pole digital filters, which are mathematically equivalent to a physical Spring-Damper-Mass system.
A significant challenge in integrating these filters into a neural network is ensuring stability. Recursive filters can "blow up" if their parameters are not carefully constrained. We derived specific stability conditions by re-parameterizing the filters using polar coordinates and squashing functions (sigmoid and tanh), ensuring that the filter poles always remain within the unit circle. Furthermore, to maintain the physiological interpretation of "spiky" nerve commands, we applied a temporal L1 constraint to the command signals, forcing the network to be efficient and sparse in its output.
Experimental Evaluation
The model was trained using the 2008 Blizzard Challenge Database, a standard corpus containing roughly 5.7 hours of high-quality speech. We used 425 text-derived linguistic features as input and extracted ground-truth pitch ($LF_0$) using the WORLD vocoder for training.
We compared our E2E model against two baselines: the previous "Atom" model (which used fixed filters) and a state-of-the-art acoustic model.
Results and Discussion
The E2E model demonstrated significant improvements in both objective and subjective metrics:
- Objective Accuracy: The Root-Mean-Squared Error (RMSE) of the synthesized pitch dropped from 28.8 Hz in the previous Atom model to 22.3 Hz in the E2E model, nearly matching the high-end baseline of 21.3 Hz.
- Subjective Naturalness: In MUSHRA listening tests, human raters found the E2E model's output to be statistically indistinguishable from the state-of-the-art baseline. Notably, the E2E model performed significantly better on complex, multi-phrase sentences where previous GCR-based models often struggled.
An interesting observation during training was that the muscle parameters tended to cluster into specific values. This suggests the network is discovering a universal set of optimal muscle responses for human speech, rather than just overfitting to the data.
Conclusion
By combining the interpretability of physiological models with the power of end-to-end neural learning, we have developed a system that produces highly natural intonation while remaining physically plausible. Future work will explore using this model for affective speech synthesis—adjusting the trainable muscle parameters to simulate different emotional states—and investigating the underlying reasons for the observed parameter clustering.
Summary updated