TL;DR
This paper introduces a low-dimensional, physically-inspired model of the human voice source (glottis) for speech processing. By fitting this model to recorded speech, they demonstrate its effectiveness for both accurate speech encoding and for performing realistic speech transformations like time stretching and pitch shifting.
What Problem Does This Paper Address?
While physical models of speech production have advanced significantly, they are rarely used in common speech processing applications like coding or synthesis. This is largely due to a lack of compact, dynamic models that can accurately represent recorded speech and allow for robust control. Existing methods, like Linear Predictive Coding (LPC), are more signal-based than physically grounded. This paper aims to bridge this gap by proposing a physically-constrained, low-dimensional glottal model that can be fitted to speech data, enabling both efficient encoding and flexible manipulation of speech characteristics.
Methods
The core of the paper is a low-dimensional dynamic glottal model based on a mass-spring system, inspired by the Ishizaka-Flanagan models. This model generates the vocal excitation signal. A key innovation is the use of a "kernel machine component" that refines the glottal flow waveform without interfering with the underlying dynamics of the glottal model itself. This "out-of-loop" refinement is crucial for maintaining stability and achieving better control during transformations.
The authors also detail a pitch-synchronous parameter identification procedure to fit this model to time-varying speech data. This involves:
- Rough Vocal Tract Estimation: Using standard LPC analysis to get initial estimates of vocal tract formants.
- Pitch Detection: Estimating the fundamental frequency to tune the mass-spring system of the glottis.
- Glottal Pulse Generation: Using the tuned glottal model to create an excitation pulse.
- Least-Squares Fitting: Employing QR factorization to jointly estimate vocal tract parameters, ensuring the synthesized speech matches the target signal.
- Glottal Pulse Refinement: Using the kernel machine to fine-tune the glottal pulse to match the inverse-filtered target glottal pulse.
This process allows for the extraction of physically meaningful parameters that can then be manipulated for speech transformations.
Datasets
The paper uses a male voice sample speaking the Italian word "aiuola" recorded at 8KHz, 16-bit mono. No new datasets are introduced; the focus is on applying the model to existing speech recordings.
Key Insights and Ideas
- Physically-Inspired Models for Practical Applications: The paper demonstrates that low-dimensional, physically-based glottal models can be made practical for speech coding and transformation, overcoming the limitations of purely signal-based approaches.
- "Out-of-Loop" Refinement is Key: Using a kernel machine to refine the glottal flow after the main glottal dynamics have been determined (out-of-loop) is shown to be more stable and controllable for transformations than an "in-loop" approach.
- Realistic Speech Transformations: The model enables time stretching and pitch shifting that are perceptually close to the original speech, retaining speaker characteristics and intelligibility.
- Improved Coding Performance: The refined glottal model, when used for speech encoding, significantly improves performance (higher SNR, lower IS distance) compared to basic LPC or LPC with synchronized glottal pulses.
Key Results
- Copy Synthesis: With the refined glottal pulse, synthesized speech is described as "perceptually almost indistinguishable" from the target.
- Time Stretching: Transformations show few audible artifacts, and importantly, avoid the phase-related degradations common in phase vocoder or sinusoidal modeling.
- Pitch Shifting: Achieved by scaling the glottal model's oscillation frequency. While effective, a slight loss of timbre naturalness is observed as the pitch shift increases, attributed to formants not being adjusted concurrently.
- Coding Performance: Table 1 shows significant improvements with the proposed model (Gl(ref)+LS) over LPC-based methods, particularly in Segmental SNR (19.28 dB vs. 2.67 dB) and Itakura-Saito distance (0.19 vs. 0.33).
Limitations and Future Work
- Formant Tracking: The authors acknowledge that inaccuracies in formant tracking can lead to minor artifacts, especially in time stretching. They did not use formant track smoothing procedures.
- Pitch Shifting Naturalness: The observed degradation in timbre naturalness during pitch shifting is linked to the fact that vocal tract formants are not modified along with the glottal frequency.
- Future Work: The authors suggest further investigation into improving formant tracking and smoothing. They also propose exploring the control of formants for more natural pitch shifting and investigating the model's versatility for other transformations like voice quality modification. They are also interested in whether the inherent dynamical structure of their model can improve perceived naturalness compared to static glottal models.
Summary updated