Abstract
In this paper, we introduce a novel language identification system based on conformer layers. We propose an attentive temporal pooling mechanism to allow the model to carry information in long-form audio via a recurrent form, such that the inference can be performed in a streaming fashion. Additionally, we investigate two domain adaptation approaches to allow adapting an existing language identification model without retraining the model parameters for a new domain. We perform a comparative study of different model topologies under different constraints of model size, and find that conformer-based models significantly outperform LSTM and transformer based models. Our experiments also show that attentive temporal pooling and domain adaptation improve model accuracy.
Introduction
Language Identification (LangID) serves as the critical "front door" for multilingual speech systems, determining which language-specific recognition model to load. While LangID is well-established for short voice commands, long-form streaming audio (such as podcasts or YouTube videos) introduces significant challenges: high latency requirements, recency bias in traditional recurrent networks, noise robustness in varied environments, and the need for hardware efficiency on modern accelerators.
To address these, we present a high-performance streaming LangID system that combines Conformer architectures with a novel attentive temporal pooling mechanism. This approach enables real-time processing with low latency and efficient memory usage, while lightweight domain adaptation techniques allow the model to excel across different audio domains without expensive retraining.
Methodology
The Conformer Backbone
Rather than relying on standard Transformers or LSTMs, our system utilizes Conformer layers. These layers integrate the global context-modeling capabilities of self-attention with the local feature extraction of convolutions. This hybrid architecture captures both long-range dependencies and fine-grained speech sounds more effectively than previous sequential models.
Attentive Temporal Pooling
To aggregate information over long durations without the "forgetting" issues typical of LSTMs, we introduced an attentive pooling mechanism. The model learns to assign weights to every audio frame, prioritizing informative speech while down-weighting silence or background noise.
For streaming efficiency, this pooling is formulated as a set of running statistics (weighted sums and sums-of-squares). This allows the model to update its language prediction at every step using constant memory, effectively maintaining an "infinite" yet computationally efficient memory of the entire audio stream.
Domain Adaptation
Training a large-scale model on 65 languages is computationally intensive. To adapt to new domains (e.g., moving from Voice Search to YouTube) without full retraining, we evaluated two methods:
- Prior Replacement: Adjusting output probabilities based on the known language distribution of a specific domain.
- Output Transform: Adding a minimal learnable layer (approximately 130 parameters) to tweak final probabilities for a target domain.
Experimental Results
Dataset and Setup
The model was trained to distinguish 65 languages using two massive Google datasets: short Voice Queries (avg. ~3.3s) and Long-form YouTube Audio (avg. ~20.7m). Training sets ranged from 1 million to 20 million utterances per language, utilizing Multi-Style Training and SpecAugment for robustness.
Performance Comparison
Across all model sizes—Small (~7MB), Medium (~30MB), and Large (~120MB)—Conformers consistently outperformed LSTMs and Transformers. As shown in the table below, the Small Conformer outperformed a similarly sized LSTM by nearly 7% absolute accuracy on voice queries.
The "Large" Conformer achieved 89.58% accuracy on voice queries and 79.22% on long-form speech. Furthermore, the Conformer models required significantly fewer floating-point operations (GFLOP/s) than Transformers to achieve superior results, making them ideal for high-load server or edge deployment.
Conclusion
Our proposed Conformer-based LangID system with attentive temporal pooling provides a stable, high-accuracy solution for streaming long-form audio. While highly effective, future work will explore multi-head attention for pooling and address "code-switching" scenarios where speakers transition between multiple languages within a single utterance.
Summary updated