Abstract
While text-based language models have achieved remarkable fluency, generative modeling in the speech domain remains challenging due to the high data requirements of raw audio processing. This paper explores a generative speech language model that operates on sub-word units—syllables and phonemes—rather than raw audio frames. By incorporating auxiliary text-based embeddings and articulatory features through multitask learning, the proposed model learns to generate speech with recognizable linguistic structure even when trained on significantly smaller datasets than current state-of-the-art "textless NLP" models.
Introduction
Recent advancements in Large Language Models (LLMs) like GPT-4 have demonstrated the power of text-based generative modeling. However, extending this success to the speech domain is non-trivial. Current approaches generally fall into two categories: Text-to-Speech (TTS) systems, which are conditioned on text and thus not autonomous language models, and "textless NLP" models like GSLM, which attempt to learn directly from raw audio. The latter often require thousands of hours of data because raw audio frames (typically 5-10ms) lack inherent linguistic structure. This research proposes a middle ground: using sub-word linguistic units as the fundamental building blocks for speech language modeling.
Methodology
The core of the proposed system is an LSTM-based encoder-decoder architecture inspired by the Continuous Bag of Words (CBOW) model. Instead of predicting the next audio frame, the model predicts the next sub-word unit (syllable or phoneme) based on a context of four preceding units.
Architecture and Multitask Learning
As shown in Figure 1, the model processes melspectrograms of context units through an LSTM encoder to extract 256-dimensional representations. These are concatenated into a latent vector ($z$), which the LSTM decoder uses to predict the melspectrogram frames of the subsequent unit. To enhance linguistic awareness, the authors implemented several strategies:
- Multitask Learning (MTL): An auxiliary classifier predicts the Panphon representation (a 66-dimensional vector of articulatory features) of the target unit.
- Auxiliary Text LMs: The latent vector $z$ is augmented with embeddings from pre-trained text-based LMs (LSTM-LM or RoBERTa) to provide external linguistic knowledge.
- Top-line Benchmark: A version of the model is provided with ground-truth articulatory features to establish an upper bound for performance.
Experimental Setup
The model was trained on the LJSpeech dataset, a single-speaker English corpus. After removing silences and excessively long units, the training set consisted of approximately 18 hours of speech. For the auxiliary text models, a subset of English Wikipedia (~30 million tokens) was used, ensuring the vocabulary matched the speech units.
Results and Discussion
The models were evaluated using Mean Opinion Score (MOS) from human listeners and linguistic prediction accuracy.
The "Babbling" Baseline
Initial experiments revealed that without linguistic guidance, the model tends to "babble," over-predicting common sounds like the "ax" phoneme. This is attributed to the highly skewed distribution of syllables in natural language, similar to a text model over-predicting the word "the."
Performance Comparison
Human evaluation (MOS) showed that the baseline synthesis achieved a score of 3.3. Incorporating Panphon-based MTL improved this to 3.4. Interestingly, adding frozen text embeddings from RoBERTa or an LSTM-LM actually decreased the MOS (to 2.9 and 2.8, respectively), likely due to the difficulty of integrating high-dimensional frozen features into a relatively small speech model. The "Top-line" model, which received ground-truth linguistic information, achieved the highest score of 3.7.
As illustrated in Figure 6, the linguistic prediction capabilities vary significantly. The Top-line model (b) shows a strong diagonal, indicating near-perfect prediction when ground-truth features are available, whereas the MTL-enhanced model (c) shows improved structure over the baseline (a), particularly in distinguishing vowel classes.
Key Insights
- Linguistic Priors are Crucial: The gap between the baseline and the Top-line model suggests that the primary bottleneck in speech LMs is not audio synthesis quality, but the accuracy of the underlying linguistic prediction.
- Metric Mismatch: Traditional acoustic metrics like Mean Squared Error (MSE) do not correlate well with human perception. The authors suggest that text-based metrics (perplexity, next-token accuracy) may be better proxies for speech quality in generative contexts.
Conclusion and Future Work
This work demonstrates that sub-word units allow for the development of speech language models with significantly less data than frame-based approaches. While the current model generates linguistically structured "babbling," future work will focus on scaling to multi-speaker datasets and improving long-term coherence to achieve true fluency in the speech domain. Better methods for fusing pre-trained text knowledge with acoustic encoders remain a priority for future research.
Summary updated