Introduction
Code-mixing—the practice of alternating between two or more languages within a single conversation or sentence—is a linguistic hallmark of multilingual societies. For instance, "Hinglish" (Hindi-English) is widely spoken across India. However, developing high-quality Text-to-Speech (TTS) systems for code-mixed speech presents a significant data challenge. Conventional models typically require extensive recordings from a "bilingual unicorn": a single speaker proficient in both languages.
Standard deterministic TTS models, such as Tacotron, often struggle with the abrupt transitions found in code-mixed text because they lack exposure to these specific language pairings during training. To address this, researchers have developed VACONDA (Variational Attention based CONtrolled Disentanglement using Articulatory priors), a system designed to generate natural code-mixed speech using only separate monolingual corpora.
Methodology: The VACONDA Framework
The core innovation of VACONDA lies in its ability to learn a language-agnostic representation of speech by leveraging universal physical constraints.
1. Variational Attention
Unlike standard models that use deterministic "soft attention" for alignment, VACONDA employs variational attention. By introducing a stochastic latent variable ($z$), the model can factorize the alignment process. This allows the system to disentangle language-specific acoustic features from the underlying phonetic structure, facilitating better generalization to unseen language combinations.
2. Articulatory Priors as Regularizers
A common failure mode in Variational Autoencoders (VAEs) is "posterior collapse," where the decoder ignores the latent code. VACONDA prevents this by replacing simple Gaussian priors with a bank of discrete distributions based on eight articulatory dimensions. These dimensions represent the physical mechanics of human speech production, such as "vowel height," "lip rounding," and "place of articulation."
Because the physical way humans produce sounds is universal across languages, forcing the model to conform to these articulatory constraints creates a common latent space. This "physics-based" disentanglement ensures that the encoder focuses on universal phonetic information rather than language-specific "nuisance factors."
3. System Architecture
As shown in the architecture diagram, the system consists of:
- Encoder: A Tacotron-style encoder that processes input text.
- VACONDA Layer: The variational attention mechanism that applies articulatory priors.
- Decoder: A WaveNet-based decoder conditioned on speaker embeddings to ensure voice consistency across language switches.
Experimental Setup and Results
The model was trained on monolingual datasets for Hindi, Telugu, and Marathi, with English used as a "scaffolding" language to bridge the gap between speakers. Evaluation was performed on a Multilingual (ML) test set containing realistic code-mixed sentences.
The performance was measured using Mean Opinion Score (MOS) for naturalness on a scale of 1 to 5. VACONDA was compared against a traditional baseline (Clustergen) and a vector-quantized model (VQTacotron).
| Configuration | Clustergen | VQTacotron | VACONDA |
|---|---|---|---|
| Tel-Eng (Female) | 3.6 | 3.9 | 4.1 |
| Mar-Eng (Male) | 3.7 | 4.0 | 4.0 |
| Mar-Eng (Female) | 3.4 | 3.9 | 4.0 |
VACONDA consistently matched or outperformed the state-of-the-art VQTacotron, showing particularly strong gains in naturalness for female voices. Qualitatively, the model produced smoother transitions at language boundaries, avoiding the irregular duration shifts common in other methods.
Conclusion and Future Work
VACONDA demonstrates that articulatory priors can effectively regularize latent spaces for cross-lingual tasks. By grounding the model in the physical reality of speech production, it achieves high-quality code-mixed synthesis without requiring bilingual training data.
Future research will focus on improving prosodic control to reduce "stylized" accents in the synthesized English and expanding phonetic coverage to include complex linguistic variants like geminate consonants.
Summary updated