When humans speak, they seamlessly transition between dozens of phonemes, modulating pitch, duration, and vocal tract shape. To replicate this computationally, early speech synthesis systems relied on a straightforward assumption: to synthesize a sound, you must first record it.
However, when building speech systems under real-world constraints—such as low-resource languages, legacy voice recordings, or one-shot voice conversion—this assumption breaks down. Systems frequently encounter a phonetic coverage gap, where the target speaker or training corpus lacks specific phonemes, syllables, or co-articulatory transitions. Understanding how synthesis models fail under incomplete phonetic coverage, and how modern architectures bypass these data constraints, is a central challenge in speech synthesis.
The Root of the Problem: Unit Granularity and Combinatorial Explosion
To understand why phonetic gaps occur, consider traditional concatenative speech synthesis (CSS). These systems slice pre-recorded audio into discrete units and glue them together to form new words. As Alan Black notes in Speech synthesis for educational technology, concatenative systems are highly dependent on the phonetic coverage of their underlying databases, making them exceptionally brittle when encountering rare or unseen units.
The severity of this brittleness depends heavily on the granularity of the chosen speech unit:
- Syllable-Level Units: Slicing audio at the syllable level preserves natural internal transitions. However, as Parminder Singh et al. demonstrate in the development of a Punjabi Text-To-Speech Synthesis System, a syllable dictionary rarely covers the entire vocabulary of a language. Out-of-vocabulary syllables or missing contextual variations fail to account for co-articulation effects, leading to choppy or impossible synthesis.
- Phoneme-Level Units: Moving to smaller units like phonemes reduces the vocabulary size but introduces a combinatorial explosion of transitions (diphones or triphones). As explained by Wael Hamza et al. in On building a concatenative speech synthesis system from the blizzard challenge speech databases, limited data means the system lacks sufficient coverage of these phonetic transitions, resulting in unnatural, disjointed audio.
This bottleneck is particularly visible when migrating legacy systems. In From pre-recorded prompts to corporate voices: on the migration of interactive voice response applications, Siegfried Kunzmann illustrates that application-specific datasets (like automated banking prompts) lack the phonetic variety required for general-purpose speech synthesis. For naturalness, the variety of phonemes matters far more than the total volume of recordings.
To maximize flexibility, researchers began shrinking unit sizes even further. Zhen-Hua Ling et al. proposed an alternative in HMM-based unit selection using frame sized speech segments, selecting tiny 5ms frame-sized units via probabilistic criteria. Slicing the database at a sub-phonetic level allows the system to recombine micro-segments dynamically, maximizing the utility of a phonetically restricted database.
How Gaps Manifest in Modern Cross-Modal and One-Shot Tasks
The phonetic coverage gap is not unique to old-school concatenative systems; it remains a fundamental hurdle for modern deep learning architectures, cross-modal synthesis, and one-shot voice conversion.
One-Shot Voice and Singing Conversion
In one-shot voice conversion, the goal is to mimic a target speaker's voice using only a few seconds of reference audio. If a source speaker utters a phoneme that does not exist in that brief target sample, how does the model translate it?
In $k$-nearest neighbor ($k\text{NN}$) based voice conversion, the system searches the target audio pool for the closest acoustic match. As Yang Li et al. observe in Phoneme Hallucinator: One-shot Voice Conversion via Set Expansion, a short reference clip lacks the necessary sounds to match diverse source inputs, causing a sharp spike in word error rates. Similarly, in Neural Concatenative Singing Voice Conversion: Rethinking Concatenation-Based Approach for One-Shot Singing Voice Conversion, Binzhu Sha et al. show that a lack of phonetic diversity in the target matching pool forces the system to substitute inappropriate segments, creating audible artifacts and "timbre leakage" where the source identity bleeds into the output.
Cross-Modal Synthesis and Eavesdropping
The problem extends to systems that reconstruct speech from non-audio modalities:
- Video-to-Speech: Models that generate audio from silent lip movements struggle when trained on limited paired datasets. Triantafyllos Kefalas et al. point out in Large-scale unsupervised audio pre-training for video-to-speech synthesis that limited corpora fail to capture intricate phoneme transitions, leading to low word accuracy.
- Speech-Driven Face Reenactment: Conversely, when driving a video using audio, Yuta Nakashima et al. demonstrate in Speech-driven Face Reenactment for a Video Sequence that if a specific phoneme or mouth shape is missing from the source video dataset, the system cannot accurately reproduce the corresponding visual speech movements.
- Acoustic Eavesdropping: Even security exploits face this limitation. In AccEar: Accelerometer Acoustic Eavesdropping with Unconstrained Vocabulary, Pengfei Hu et al. show that reconstructing speech from smartphone accelerometer vibrations requires a training dataset encompassing all possible sounds to achieve true unconstrained vocabulary reconstruction.
Whether dealing with assistive hand-to-speech converters restricted to narrow consonant subsets (Aki Kunikoshi et al.), real-time singing synthesis tools limited to vowels (Paul Chan et al.), or low-resource language modeling (Fei He et al.), the core challenge is identical: the model cannot synthesize what it has never observed.
Algorithmic Solutions: Bridging the Gap
To overcome these limitations, researchers have developed techniques to "hallucinate" missing data, leverage physical constraints, or utilize hybrid architectures.
1. Articulatory Control
If an acoustic recording of a vowel does not exist, can we synthesize it by modeling the human body? Human speech is constrained by physical anatomy—the positioning of the tongue, lips, and jaw.
In Vowel Creation by Articulatory Control in HMM-based Parametric Speech Synthesis, Zhen-Hua Ling et al. utilized a Multiple Regression Hidden Markov Model (MRHMM) to map physical articulatory movements to acoustic outputs. By moving the virtual "tongue" and "lips" into positions the training data had never explicitly seen, they successfully synthesized entirely new vowels (such as the English /ʌ/) with high clarity. This physical grounding bypasses acoustic data scarcity, though it requires highly accurate vocal tract models to avoid the missing feature gaps common in synthetic tract approximations (Matthew Caren et al.).
2. Phoneme Hallucination and Set Expansion
Instead of physically modeling the vocal tract, can we mathematically infer what a speaker would sound like pronouncing an unseen phoneme?
Yang Li et al. solved the one-shot voice conversion bottleneck in Phoneme Hallucinator by using a set expansion framework. The system takes a tiny 3-second sample of a target speaker and "hallucinates" a diverse set of synthetic phoneme representations. By expanding this sparse sample into a rich, complete library of virtual sounds, the downstream $k\text{NN}$ search always finds an appropriate phonetic match, drastically reducing word error rates.
[3-Second Target Audio] ──> [Phoneme Hallucinator] ──> [Rich Synthetic Sound Library]
│
[Source Speech Input] ──> [kNN Feature Matching] <─────────┘
│
▼
[High-Fidelity Conversion]
3. Large-Scale Unsupervised Pre-Training
When paired data (e.g., video-to-speech or text-to-speech pairs) is limited, models fail to learn the underlying grammar of speech transitions. To fix this, Triantafyllos Kefalas et al. decoupled the learning of speech structure from the cross-modal task. By pre-training an audio decoder on over 3,500 hours of unsupervised, audio-only data, the model thoroughly learns phoneme transitions and prosody. When later fine-tuned on a limited paired video dataset, the model relies on its pre-trained acoustic priors to smoothly fill in the phonetic coverage gaps.
4. Hybrid Fallbacks and Donor Databases
When working with legacy data or hybrid synthesis systems, explicit architectural fallbacks can maintain quality. Xiao Zhou et al. introduced UnitNet-Based Hybrid Speech Synthesis, which bridges statistical parametric synthesis and concatenative synthesis. Rather than restricting generative fallback mechanisms to specific phonetic categories (like voiced phones), they expanded these mechanisms across all phone types, ensuring that if a concatenative unit is missing, a generative model seamlessly fills the gap. Similarly, Siegfried Kunzmann demonstrated that morphing the output of a high-quality "donor" database to match a target speaker's spectral characteristics can successfully enrich phonetically starved legacy systems.
By shifting from rigid acoustic lookup tables to flexible frame-level selection, articulatory simulation, and neural phoneme hallucination, modern speech synthesis bypasses the physical limitations of source data, ensuring fluid communication even when the data itself is incomplete.
Key Foundational Work
The CMU Arctic speech databases (2004) — contributed 2 concept elements to this direction.
The effect of tonal context on cantonese concatenative speech synthesis (2002) — contributed 1 concept elements to this direction.
A database design for a TTS synthesis system using lexical diphones (2004) — contributed 1 concept elements to this direction.