Attentive Temporal Pooling for Conformer-based Streaming Language Identification in Long-form Speech

Quan Wang, Yang Yu, Jason Pelecanos, Yiling Huang, Ignacio López Moreno

Abstract

In this paper, we introduce a novel language identification system based on conformer layers. We propose an attentive temporal pooling mechanism to allow the model to carry information in long-form audio via a recurrent form, such that the inference can be performed in a streaming fashion. Additionally, we investigate two domain adaptation approaches to allow adapting an existing language identification model without retraining the model parameters for a new domain. We perform a comparative study of different model topologies under different constraints of model size, and find that conformer-based models significantly outperform LSTM and transformer based models. Our experiments also show that attentive temporal pooling and domain adaptation improve model accuracy.

Introduction

Language Identification (LangID) serves as the critical "front door" for multilingual speech systems, determining which language-specific recognition model to load. While LangID is well-established for short voice commands, long-form streaming audio (such as podcasts or YouTube videos) introduces significant challenges: high latency requirements, recency bias in traditional recurrent networks, noise robustness in varied environments, and the need for hardware efficiency on modern accelerators.

To address these, we present a high-performance streaming LangID system that combines Conformer architectures with a novel attentive temporal pooling mechanism. This approach enables real-time processing with low latency and efficient memory usage, while lightweight domain adaptation techniques allow the model to excel across different audio domains without expensive retraining.

Methodology

The Conformer Backbone

Rather than relying on standard Transformers or LSTMs, our system utilizes Conformer layers. These layers integrate the global context-modeling capabilities of self-attention with the local feature extraction of convolutions. This hybrid architecture captures both long-range dependencies and fine-grained speech sounds more effectively than previous sequential models.

Attentive Temporal Pooling

To aggregate information over long durations without the "forgetting" issues typical of LSTMs, we introduced an attentive pooling mechanism. The model learns to assign weights to every audio frame, prioritizing informative speech while down-weighting silence or background noise.

Architecture of the Conformer-based LangID system featuring attentive temporal pooling for streaming inference.

For streaming efficiency, this pooling is formulated as a set of running statistics (weighted sums and sums-of-squares). This allows the model to update its language prediction at every step using constant memory, effectively maintaining an "infinite" yet computationally efficient memory of the entire audio stream.

Domain Adaptation

Training a large-scale model on 65 languages is computationally intensive. To adapt to new domains (e.g., moving from Voice Search to YouTube) without full retraining, we evaluated two methods:

  1. Prior Replacement: Adjusting output probabilities based on the known language distribution of a specific domain.
  2. Output Transform: Adding a minimal learnable layer (approximately 130 parameters) to tweak final probabilities for a target domain.

Experimental Results

Dataset and Setup

The model was trained to distinguish 65 languages using two massive Google datasets: short Voice Queries (avg. ~3.3s) and Long-form YouTube Audio (avg. ~20.7m). Training sets ranged from 1 million to 20 million utterances per language, utilizing Multi-Style Training and SpecAugment for robustness.

Performance Comparison

Across all model sizes—Small (~7MB), Medium (~30MB), and Large (~120MB)—Conformers consistently outperformed LSTMs and Transformers. As shown in the table below, the Small Conformer outperformed a similarly sized LSTM by nearly 7% absolute accuracy on voice queries.

Comparative study of model sizes and encoder types, demonstrating the superior accuracy and efficiency of Conformer architectures.

The "Large" Conformer achieved 89.58% accuracy on voice queries and 79.22% on long-form speech. Furthermore, the Conformer models required significantly fewer floating-point operations (GFLOP/s) than Transformers to achieve superior results, making them ideal for high-load server or edge deployment.

Conclusion

Our proposed Conformer-based LangID system with attentive temporal pooling provides a stable, high-accuracy solution for streaming long-form audio. While highly effective, future work will explore multi-head attention for pooling and address "code-switching" scenarios where speakers transition between multiple languages within a single utterance.

Summary updated

Abstract

In this paper, we introduce a novel language identification system based on conformer layers. We propose an attentive temporal pooling mechanism to allow the model to carry information in long-form audio via a recurrent form, such that the inference can be performed in a streaming fashion. Additionally, we investigate two domain adaptation approaches to allow adapting an existing language identification model without retraining the model parameters for a new domain. We perform a comparative study of different model topologies under different constraints of model size, and find that conformer-based models significantly outperform LSTM and transformer based models. Our experiments also show that attentive temporal pooling and domain adaptation improve model accuracy.
BibTeX
@misc{wang2022attentive,
  author = {Quan Wang and Yang Yu and Jason Pelecanos and Yiling Huang and Ignacio López Moreno},
  title = {{Attentive Temporal Pooling for Conformer-based Streaming Language Identification in Long-form Speech}},
  year = {2022},
  eprint = {2202.12163},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2202.12163},
}
Figure