TL;DR
This paper introduces a novel Finite-State Transducer (FST) architecture for corpus-based concatenative speech synthesis. By decoupling domain-independent synthesis costs into a "constraint kernel," their approach achieves linear scalability with corpus size, enabling efficient, real-time, and natural-sounding speech generation for conversational systems.
What Problem Does This Paper Address?
Generating natural-sounding speech from text (Text-to-Speech or TTS) is a complex task. Corpus-based concatenative synthesis, which stitches together pre-recorded speech segments, has shown great promise for naturalness. However, efficiently selecting the best segments from a large corpus and managing the computational complexity of this selection process has been a significant challenge. Previous methods often required pruning or were not easily scalable. The authors aim to create a flexible and scalable architecture that can handle the intricacies of unit selection and waveform concatenation in real-time for conversational AI systems.
Methods
The core innovation is the representation of the speech synthesis process as a series of cascaded Finite-State Transducers (FSTs). This FST-based approach allows for a unified framework, leveraging existing work in speech recognition. The key components are:
- W (Word Sequence): A trivial FST representing the input text.
- L (Lexicon): Maps words to their baseform pronunciations (phonemes).
- P (Phonological Rules): Apply rules to refine pronunciations.
- T (Transition Phones): Inserts intermediate "transition phones" between adjacent phonemes to constrain the search space and improve efficiency.
- S (Synthesis FST): This is the most novel part. It maps phone sequences to waveform segments. The authors designed a "constraint kernel" within S that encapsulates domain-independent substitution and concatenation costs. This kernel uses intermediate layers (specifically, four layers are described) to efficiently bridge source and destination speech segments. This design ensures that the FST's size scales linearly with the size of the speech corpus, avoiding the need for pruning mechanisms that would be required if all possible segment connections were explicitly modeled.
The unit selection itself is performed using a Viterbi or N-best search algorithm, similar to those used in speech recognition.
Datasets
The paper does not explicitly list specific datasets used for training or evaluation. However, it refers to "a domain-dependent corpus" and "a large speech corpus" from which waveform segments are drawn. The synthesis system is integrated into their "conversational system architecture" and has been applied to domains like "MERCURY air travel" and "weather information."
Key Insights and Ideas
- FSTs as a Unified Framework: Representing speech synthesis as an FST composition provides a clear, consistent, and flexible architecture, allowing for the reuse of techniques from speech recognition (like pronunciation modeling and search).
- Scalable Constraint Kernel: The proposed four-layer constraint kernel in the synthesis FST (S) is a crucial design element. By using equivalence classes for phones and intermediate layers, it decouples domain-independent costs and ensures that the FST's complexity grows linearly with the corpus size, making it highly scalable.
- Incorporating Substitution Costs: Unlike some previous methods that focused solely on concatenation costs, this FST architecture explicitly incorporates both substitution and concatenation costs within the FST itself.
- Transition Phones for Search Efficiency: The use of transition phones (via FST T) significantly prunes the search space, enabling real-time synthesis.
Key Results
The paper reports "preliminary success" with the new FST-based synthesis in "several constrained spoken dialogue applications." They have integrated the FST synthesizer into real-time servers within their GALAXY conversational system architecture, achieving speeds sufficient for interactive purposes. The system is capable of producing "very natural sounding speech" for their spoken dialogue domains.
Limitations and Future Work
- Acoustic Information: The current design's costs are derived solely from symbolic contextual information. It is not yet possible to introduce segment-specific costs based on acoustic information, as seen in some other systems. The authors plan to explore incorporating quantized acoustic information.
- Signal Processing: While concatenation is currently performed without signal processing, they are exploring signal processing techniques (like modifying fundamental frequency and segment duration) to reduce concatenation artifacts.
- Prosody: Prosody has been handled at the lexical level or through "shortcuts" (pre-recorded segments). They plan to investigate corpus-based prosodic generation for more natural prosody.
- Language Support: The current work is primarily focused on English. They are actively developing a Mandarin Chinese version and plan to work on Spanish and Japanese synthesizers.
Summary updated