Speech recognition systems that operate without traditional acoustic signals—such as visual speech recognition (lipreading), electromyography (EMG), and electroencephalography (EEG) brain-computer interfaces—excel in constrained environments. When a user only needs to say "yes," "no," or a few digits, these systems achieve remarkable accuracy.
However, when transitioning from a handful of commands to natural, open-vocabulary language, performance frequently degrades. This drop-off is driven by a fundamental challenge: vocabulary scalability. Mapping noisy, high-dimensional non-acoustic signals directly to a massive discrete search space of tens of thousands of words creates an exponential explosion of ambiguity and data scarcity.
To build systems capable of handling unconstrained natural language, researchers must move away from direct signal-to-word classification. Instead, they employ strategies that break down the problem into invariant sub-word units and leverage cross-modal pretraining.
The Combinatorial Explosion of Direct Mapping
To understand why large vocabularies break naive non-acoustic speech classifiers, consider the visual or neural signatures of speech. In acoustic speech, a word like "bat" has distinct spectral transitions that separate it clearly from "mat." Visually, however, the initial phonemes /b/ and /m/ are homophenes—they look identical on the lips. Neurally, the imagined execution of these similar motor patterns produces highly overlapping signals.
When a model is trained on a small, closed-set vocabulary, it can rely on subtle, idiosyncratic context clues to tell words apart. But as the vocabulary expands to thousands of words, these wide decision boundaries collapse. As noted by Arnav Kapur et al., transitioning to open-vocabulary dictation introduces severe neuromuscular decoding hurdles because the model must distinguish between thousands of highly overlapping physical or neural patterns.
Furthermore, collecting enough training data for a full lexicon is incredibly difficult. As Jinzhao Zhou et al. point out, scaling to a full lexicon requires overcoming significant data scarcity; you cannot easily record a user repeating every word in a dictionary thousands of times while wearing an EEG cap or an EMG patch. Without a way to generalize to unseen words, direct classification architectures fail.
Deconstructing Words into Invariant Sub-Units
The primary mechanism for achieving vocabulary scalability is to insert an intermediate, lower-dimensional bottleneck between the raw signal and the final text. Instead of mapping a signal directly to a word, models are designed to decode invariant, universal building blocks of language, such as phonemes, phonological categories, or discrete articulatory movements.
[Raw Non-Acoustic Signal]
│
▼
[Intermediate Bottleneck] (Phonemes, Articulatory Symbols, Features)
│
▼
[Language Model / Decoder] ──► [Open-Vocabulary Text]
This hierarchical approach fundamentally changes the complexity of the task. Instead of learning a classification space of 50,000 words, the model only needs to learn to classify a few dozen phonetic or articulatory targets.
Phonological and Phonemic Bottlenecks
In EEG-based imagined speech recognition, Pramit Saha et al. demonstrated that decoding intermediate phonological categories (such as distinguishing whether a thought sound is "nasal" or "bilabial") provides a significant performance boost over direct word classification. By organizing the feature space around these fundamental linguistic properties, the model can compose full words out of recognized sub-components, mitigating the performance gap typically seen when moving from simple phonetic units to multi-syllabic words.
A similar breakthrough occurred in visual speech recognition. Traditional lipreading models struggled to generalize beyond narrow datasets. To solve this, Brendan Shillingford et al. introduced a Vision-to-Phoneme (V2P) architecture. By training the neural network to predict phonemes rather than characters or whole words, and then passing those phonemes to a production-grade speech decoder, the system successfully scaled to an open vocabulary of unconstrained human speech.
Articulatory Discretization
Another way to create an invariant bottleneck is to map physical movements to discrete symbols. For systems tracking physical speech production without sound (such as tongue and lip movements), Arvind Balasubramanian et al. showed that representing continuous articulatory motion time-series data as a sequence of discrete symbols allows the system to scale toward real-time, whole-word recognition. Discretizing the continuous physical space prevents the model from getting lost in the infinite variations of raw muscle movement, providing a structured representation that downstream language decoders can easily parse.
Cross-Modal Anchoring and Pretraining
Even with intermediate representations, non-acoustic signals are often too noisy to train from scratch on large vocabularies. To prevent representation collapse, researchers anchor the non-acoustic feature space using more abundant data modalities, such as audio or multi-mode speech signals.
Leveraging Audio as a Teacher
During training, audio can act as a powerful regularizer for visual or neural models. For instance, Chao Sui et al. developed a framework using Multimodal Deep Boltzmann Machines (DBMs) that learns visual features by explicitly leveraging audio data during the training phase. Because the model learns the joint distribution of audio and video, it can "hallucinate" the missing acoustic features at test time when given only a silent video. This cross-modal grounding ensures that the visual features retain the discriminatory power of audio, which is essential for scaling to large vocabularies where visual cues alone are ambiguous.
Self-Supervised Brain Modeling
For brain-computer interfaces where data is exceptionally scarce, researchers use large-scale self-supervised pretraining to learn the underlying structure of brain activity before attempting word classification. Jinzhao Zhou et al. introduced the Large Brain Language Model (LBLM), which utilizes a two-stage pretraining strategy to predict future brain activity across both time and frequency domains. By mastering the dynamics of the neural signal itself in an unsupervised manner, the model creates a robust foundation that can map neural patterns to thousands of distinct linguistic units with minimal fine-tuning data.
Multi-Mode Joint Modeling
Finally, models can borrow strength from different execution modes of speech. Fei Chen et al. developed a joint modeling framework that trains a single model simultaneously on imagined speech, silent mouthing, and actual spoken speech.
[Imagined Speech] ──┐
[Silent Mouthing] ──┼─► [Shared Multi-Mode Model] ──► Robust Vocabulary Scaling
[Spoken Speech] ──┘
Because spoken speech generates clear, highly structured neural signals, the joint model uses those strong patterns to anchor and organize the faint, noisy representations of purely imagined thoughts. This shared representation space allows the system to scale its vocabulary capacity far beyond what imagined speech data alone could support.