When building multilingual models, natural language processing traditionally treats text as the ultimate ground truth. However, text is an arbitrary, language-specific projection of human speech. A single character can represent vastly different sounds across borders, and entirely different scripts—such as Latin, Cyrillic, Devanagari, or Arabic—completely isolate languages that might otherwise share deep acoustic roots, cognates, or structural patterns. This creates an orthographic bottleneck that severely limits cross-lingual knowledge transfer, particularly for low-resource languages.
To bypass this barrier, a powerful research direction has emerged: mapping diverse scripts and speech signals into a universal phonetic intermediate representation. By converting text or audio into a standardized phonetic alphabet—such as the International Phonetic Alphabet (IPA) or X-SAMPA—models can operate in a language-independent space. In this space, linguistic tokens are organized not by arbitrary spelling conventions, but by how they actually sound.
Bypassing the Script Barrier
The core intuition behind a unified phonetic representation is simple: while writing systems diverge, the physical mechanics of human speech are universal. When we strip away surface-level orthography and represent words as sequences of universal speech sounds, models can recognize cross-lingual similarities that were previously obscured.
[English Text] -> "night" ---------\
+--> IPA: /naɪt/ --> [Shared Embedding Space]
[German Text] -> "Nacht" ---------/
This approach is highly effective for cross-lingual transfer between languages with different writing systems. For instance, Aditi Chaudhary et al. demonstrated in Adapting Word Embeddings to New Languages with Morphological and Phonological Subword Representations that representing words as phoneme sequences using the IPA allows models to discover phonological relationships across different scripts. This enables the transfer of word embeddings from high-resource to low-resource languages without requiring expensive bilingual dictionaries.
The same mechanism applies to modern large language models (LLMs), which are heavily biased toward Latin scripts due to their training data. To address this, Hoang H Nguyen et al. introduced PhoneXL in Enhancing Cross-lingual Transfer via Phonemic Transcription Integration, showing that integrating IPA transcriptions directly into a model's vocabulary provides a consistent ground truth that outperforms language-specific romanization schemes. In subsequent work, Hoang H Nguyen et al. developed Mixed-ICL in Prompting with Phonemes: Enhancing LLMs' Multilinguality for Non-Latin Script Languages, proving that prompting LLMs with phonemic transcriptions bridges the script gap, allowing the model to seamlessly transfer semantic knowledge from high-resource Latin text to non-Latin target languages.
A Common Ground for Speech and Text
Phonetic representations do more than just bridge different scripts; they also bridge different modalities. Continuous audio signals are highly variable and noisy, while text tokens are discrete but script-bound. Phonemes serve as the ideal intermediate representation because they are both discrete (like text) and directly tied to acoustic properties (like speech).
[Continuous Audio] ----> [Acoustic Encoder] ---\
+--> [Universal Phonetic Space]
[Written Text] --------> [G2P Conversion] -----/
By forcing speech and text into a shared phonetic space, models can align the two modalities with minimal paired data. Colin Leong explored this in Phone-ing it in: Towards Flexible Multi-Modal Language Model Training by Phonetic Representations of Data, proposing a framework where both raw audio and text are mapped into a shared symbolic IPA space. This multi-modal alignment allows speech models to learn linguistic structures from text data, and vice versa, boosting performance on downstream tasks like Named Entity Recognition (NER).
In speech translation, the authors of FST: the FAIR Speech Translation System for the IWSLT21 Multilingual Shared Task utilized phonetic units instead of standard subword tokens. This strategy leverages the inherent relationship between spoken sounds and phonetic symbols to improve alignment between audio inputs and translation targets, enhancing cross-lingual transfer across diverse writing systems.
Similarly, for automatic speech recognition (ASR), Yang Yuhang et al. demonstrated in Bridging Speech and Text: Enhancing ASR with Pinyin-to-Character Pre-training in LLMs that using a discrete phonetic intermediate representation (like Pinyin) allows LLMs to master the relationship between pronunciation and semantics before they ever have to process continuous, high-frequency audio waves.
Scaling Speech Synthesis and Recognition
In Text-to-Speech (TTS) and ASR systems, a universal phonetic space acts as a powerful regularizer. Instead of training separate, isolated models for every language, a single model can treat all human speech as variations within a single continuous phonetic framework.
For speech synthesis, the lack of expert-curated pronunciation dictionaries for low-resource languages has long been a bottleneck. Sunayana Sitaram et al. addressed this in Universal grapheme-based speech synthesis by mapping standardized Unicode character codes directly to a universal phonetic alphabet. This allows a single speech synthesis framework to support thousands of languages by leveraging shared phonetic properties across different language families.
This shared space also enables cross-lingual voice cloning. As shown by Mengnan Chen et al. in Cross-Lingual, Multi-Speaker Text-To-Speech Synthesis Using Neural Speaker Embedding, mapping multiple languages to a unified phonetic representation allows a TTS model to share acoustic knowledge across linguistic boundaries. Consequently, the system can synthesize speech in a language the original speaker does not know, treating the input as a trajectory through a single, continuous phonetic space while preserving speaker identity.
On the speech recognition side, Ming Tu et al. demonstrated in Language-universal Phonetic Encoder for Low-resource Speech Recognition that pre-training an acoustic encoder to predict universal phonetic symbols creates a language-agnostic representation. When the model encounters a new, data-scarce language, it does not need to learn how to decode speech from scratch; it simply maps the new language's vocabulary onto the universal phonetic currency it has already mastered.
Ultimately, this research direction demonstrates that language-specific orthographies are merely surface variations. By anchoring models in a universal phonetic space, speech and language technologies become truly multilingual, generalizing patterns across scripts, modalities, and language families.