Abstract
This paper presents a unified model to perform language and speaker recognition simultaneously. The model is based on a multi-task recurrent neural network where the output of one task is fed as the input of the other, leading to a collaborative learning framework. This approach allows both language and speaker recognition to improve by borrowing information from each other. Our experiments demonstrate that the multi-task model outperforms task-specific models on both tasks.
Introduction
In speech processing, Language Recognition (LRE) and Speaker Recognition (SRE) have traditionally been treated as separate silos. LRE researchers often view speaker variation as "noise" to be filtered out, while SRE researchers struggle with "language mismatch," where models fail when a speaker switches languages.
This paper argues that this separation is artificial and unlike human audition. Humans recognize speakers and languages simultaneously, and knowing one often aids in identifying the other. We propose a unified architecture where these two correlated factors are modeled together in a collaborative learning framework.
Methods
The system is built using Long Short-Term Memory (LSTM) networks, which are well-suited for sequential speech data. The core innovation is the Collaborative Learning architecture, which moves beyond traditional multi-task learning by propagating specific task outputs back into the network in real-time.
As shown in the architecture above, the model propagates the specific outputs (the "r-vector" and "p-vector") of one task into the non-linear processing units (gates) of the other task at the next time step. This creates a dynamic feedback loop where linguistic knowledge informs speaker identification, and speaker-specific traits help normalize the signal for language recognition.
Experimental Setup
The study utilized two primary databases to create a bilingual environment:
- WSJ (Wall Street Journal): An English-language corpus.
- CSLT-C300: A Chinese-language corpus.
The evaluation included both Full-length utterances and Short-length segments (only 1 second of speech) to test robustness under data-constrained conditions. We compared our LSTM-based "r-vectors" against the traditional "i-vector" gold standard.
Key Results and Insights
Performance Gains in LRE
The most striking finding was the improvement in Language Recognition. The SRE component effectively "normalizes" the signal, allowing the LRE component to focus purely on linguistic patterns. In full-length tests, the number of identification errors (IDE) dropped from 29 in the baseline to nearly zero in several collaborative configurations.
Robustness on Short Utterances
As seen in the results table, while i-vector models perform well on long audio, the LSTM-based "r-vectors" are significantly more effective for very short (1-second) clips. For example, in the short-length test, the r-vector/Softmax configuration achieved an IDE of 129, compared to 2270 for the i-vector/Cosine baseline.
Gate Sensitivity
The research also explored which parts of the LSTM benefited most from cross-task feedback. Results indicated that the output gate was the most effective location for feeding information into the SRE task, while the forget gate was more beneficial for the LRE task.
Conclusion
Collaborative learning represents a significant step toward more human-like machine audition. By sharing knowledge "on-line" during processing, the model achieves superior performance, particularly in language recognition and short-duration speaker identification. Future work will focus on scaling this framework to more diverse datasets and a larger number of languages.
Summary updated