TL;DR
This paper introduces a novel "one-model" approach to speech recognition (SR) and synthesis (SS) that leverages a shared articulatory movement HMM. This unified model promises more efficient and integrated speech processing systems, enabling functionalities like phoneme-to-phoneme conversion.
What Problem Does This Paper Address?
Traditional speech recognition and synthesis systems often operate independently, using different feature representations and models. Speech recognition typically relies on spectral features (like MFCCs) that can be sensitive to variations in speaker, noise, and recording conditions. Speech synthesis, on the other hand, often uses speaker-specific models. This disconnect makes it challenging to build systems that can seamlessly switch between understanding and generating speech, or to develop advanced features like pronunciation correction. The authors aim to bridge this gap by proposing a single, unified model based on articulatory movements, which are hypothesized to be more fundamental to speech production and perception.
Methods
The core of this work is the "one-model" framework that uses a common set of Articulatory Movement Hidden Markov Models (AM HMMs).
- Articulatory Feature (AF) Extraction: Input speech is first processed to extract articulatory features. This involves converting spectral features into local features (LFs) and then using a three-stage multi-layer neural network (MLN) to extract AFs. These AFs represent the movements of the vocal tract.
- Speech Recognition (SR) Engine: The extracted AFs are fed into the AM HMMs, which are trained to classify phonemes. This approach aims for speaker-invariant recognition.
- Speech Synthesis (SS) Engine: The same AM HMMs are used to generate an AF sequence. This sequence is then converted into vocal tract parameters (PARCOR parameters) using a speaker-adapted MLN. Finally, a PARCOR synthesizer generates the speech signal.
- Phoneme-to-Phoneme Conversion: A unique capability of this one-model approach is the ability to swap AFs between phonemes. For example, the AF sequence for phoneme 'A' can be replaced with the AF sequence for phoneme 'B' to generate a modified speech output.
Datasets
- MLN Training Data: A subset of the Acoustic Society of Japan (ASJ) Continuous Speech Database (4,503 sentences from 30 male speakers) was used for training the MLNs in the AF extractor.
- HMM Training Data: The JNAS (Japanese Newspaper Article Sentences) database was used for training the AM HMMs. Two sets were used: one with 5,000 sentences from 33 male speakers for HMMs and another with 5,000 sentences for the AF-PARCOR converter in the SS engine.
- Test Data: 2,719 JNAS sentences from 17 male speakers were used for evaluating the SR performance.
Key Insights and Ideas
- Unified Model for SR and SS: The most significant idea is the unification of SR and SS under a single articulatory movement model. This suggests a deeper connection between speech perception and production than previously exploited in many systems.
- Speaker-Invariant Articulatory Features: The AFs extracted appear to be more speaker-invariant than traditional spectral features like MFCCs, as demonstrated by consistent SR performance across different numbers of enrolled speakers.
- Potential for Pronunciation Training: The ability to perform phoneme-to-phoneme conversion by swapping AFs opens up exciting possibilities for interactive pronunciation training and correction.
Key Results
- Speech Recognition: The proposed AF-based HMMs achieved comparable or better phoneme recognition rates than MFCC-based HMMs, even with limited training data (single speaker, single mixture). The performance was robust to variations in the number of enrolled speakers and HMM mixtures.
- Speech Synthesis: The MOS (Mean Opinion Score) for synthetic speech showed improvement with adaptation. While initial quality without adaptation was moderate, using just two sentences for adaptation yielded a noticeable improvement, with further gains from thirty-two sentences. Spectrogram analysis also showed that adaptation improved the similarity of synthetic speech to the target speaker.
- Phoneme-to-Phoneme Conversion: Experiments showed promising results in converting phonemes like /b/ to /d/, /p/ to /t/, and /m/ to /n/, with subjects able to distinguish the converted phonemes in an ABX listening test.
Limitations and Future Work
- Speech Synthesis Quality: While adaptation improved synthetic speech quality, the authors acknowledge that further improvement is needed. They suggest exploring CELP (Code-Excited Linear Prediction) approaches for the voice source.
- Phoneme-to-Phoneme Conversion Accuracy: The phoneme-to-phoneme conversion, while promising, still requires further refinement.
- Future Work: The authors plan to improve SS voice quality and explore the implementation of SR engines based on articulatory movement HMMs.
Summary updated