For decades, speech technology relied on a specialized bridge between written text and spoken sound: the pronunciation lexicon. This expert-curated dictionary maps words (graphemes) to their constituent sounds (phonemes). However, as Bell et al. argue in The UEDIN ASR systems for the IWSLT 2014 evaluation, this reliance creates a significant barrier to deploying systems in new languages or dialects where such resources do not exist.
A growing body of research proposes a simpler alternative: bypassing phonemes entirely and mapping audio directly to graphemes. By treating characters as the fundamental units of sound, researchers are building "lexicon-free" systems that are more flexible, easier to scale, and often more accurate than their phonetic predecessors.
The Phonetic Bottleneck
Traditional speech recognition and synthesis pipelines are modular. To recognize the word "knight," a system first identifies the sounds /n/, /aɪ/, /t/, then looks up those sounds in a lexicon to find the matching spelling. This approach assumes that phonemes are the "true" internal representation of speech.
But phonemes are human constructs, and lexicons are brittle. They struggle with out-of-vocabulary (OOV) words, spontaneous speech disfluencies, and the diverse accents of non-native speakers. As Prabhavalkar et al. demonstrate in No Need for a Lexicon? Evaluating the Value of the Pronunciation Lexica in End-to-End Models, end-to-end (E2E) models can internalize these relationships directly. By predicting graphemes from audio, these models jointly optimize acoustic and linguistic representations within a single neural network.
Learning Phonetics from Spelling
The intuition behind grapheme-based modeling is that deep neural networks are powerful enough to discover the latent rules of pronunciation from data. In Towards End-To-End Speech Recognition with Recurrent Neural Networks, Graves et al. showed that models using Connectionist Temporal Classification (CTC) can learn to transcribe audio at the character level without any pre-calculated alignments or phonetic maps.
This works because the model learns context. For example, in English, the letter "c" sounds different in "cat" versus "city." A context-aware model learns these patterns implicitly. Le et al. took this further in From Senones to Chenones: Tied Context-Dependent Graphemes for Hybrid Speech Recognition, introducing "chenones"—context-dependent graphemes that allow hybrid systems to outperform traditional phone-based models by removing the expert-curated phonetic bottleneck.
Robustness to the Unknown
One of the primary advantages of lexicon-free modeling is how it handles the "edges" of language.
- Out-of-Vocabulary Words: Traditional systems fail when they encounter a word not in their dictionary. In contrast, the authors of Lexicon-Free Conversational Speech Recognition with Neural Networks show that character-level decoders can simply "spell out" unknown words phonetically, maintaining robustness in spontaneous conversation.
- Non-Native Speech: Language learners often pronounce words based on how they are spelled rather than standard phonetic rules. Knill et al. found in Non-Native Children's Automatic Speech Recognition: The INTERSPEECH 2020 Shared Task ALTA Systems that graphemic lexicons actually outperform phonetic ones for low-proficiency speakers, as the letter-based model better matches the learner's spelling-influenced speech.
- Implicit Rules: In languages like Hindi, certain vowels are dropped depending on their position in a word. Jyothi et al. demonstrated in Improved hindi broadcast ASR by adapting the language model and pronunciation model using a priori syntactic and morphophonemic knowledge that by including word-boundary markers, graphemic models can implicitly learn these complex morphophonemic rules without an explicit dictionary.
Scaling to the "Data Desert"
For many of the world's languages, building a phonetic dictionary is an impossible prerequisite. Grapheme-based modeling provides a "from-scratch" path for low-resource languages. The BibleTTS project utilized raw audio and text graphemes to create high-fidelity alignment for Sub-Saharan African languages where no phonetic tools existed. Similarly, Shi et al. showed in Leveraging End-to-End ASR for Endangered Language Documentation: An Empirical Study on Yolóxochitl Mixtec that E2E Transformer models can outperform traditional systems in documenting endangered languages by bypassing the need for manual phonetic transcription.
Beyond Recognition: Synthesis
The shift toward graphemes is also transforming Text-to-Speech (TTS). Traditional TTS requires a Grapheme-to-Phoneme (G2P) front-end, which adds engineering complexity. Modern systems like E2 TTS by Eskimez et al. use large-scale training to map raw character inputs directly to audio. This "embarrassingly easy" approach matches the naturalness of complex phonetic systems while being far more robust to different speech rates and speaker characteristics.
By removing the intermediate symbolic layer of phonemes, researchers have found that models can more effectively learn the direct, fluid relationship between the written word and the human voice.
Key Foundational Work
A unified framework for large vocabulary speech recognition of mutually unintelligible Chinese "regionalects" (2004) — contributed 2 concept elements to this direction.
Development of a conversational telephone speech recognizer for Levantine Arabic (2005) — contributed 1 concept elements to this direction.