Collaborative Learning for Language and Speaker Recognition

Lantian Li, Zhiyuan Tang, Dong Wang, Yang Feng, Shiyue ZhangarXiv 2016 · OpenReview

Abstract

This paper presents a unified model to perform language and speaker recognition simultaneously. The model is based on a multi-task recurrent neural network where the output of one task is fed as the input of the other, leading to a collaborative learning framework. This approach allows both language and speaker recognition to improve by borrowing information from each other. Our experiments demonstrate that the multi-task model outperforms task-specific models on both tasks.

Introduction

In speech processing, Language Recognition (LRE) and Speaker Recognition (SRE) have traditionally been treated as separate silos. LRE researchers often view speaker variation as "noise" to be filtered out, while SRE researchers struggle with "language mismatch," where models fail when a speaker switches languages.

This paper argues that this separation is artificial and unlike human audition. Humans recognize speakers and languages simultaneously, and knowing one often aids in identifying the other. We propose a unified architecture where these two correlated factors are modeled together in a collaborative learning framework.

Methods

The system is built using Long Short-Term Memory (LSTM) networks, which are well-suited for sequential speech data. The core innovation is the Collaborative Learning architecture, which moves beyond traditional multi-task learning by propagating specific task outputs back into the network in real-time.

The Collaborative Learning architecture showing two parallel LSTMs (LRE and SRE) with cross-task feedback loops.

As shown in the architecture above, the model propagates the specific outputs (the "r-vector" and "p-vector") of one task into the non-linear processing units (gates) of the other task at the next time step. This creates a dynamic feedback loop where linguistic knowledge informs speaker identification, and speaker-specific traits help normalize the signal for language recognition.

Experimental Setup

The study utilized two primary databases to create a bilingual environment:

  • WSJ (Wall Street Journal): An English-language corpus.
  • CSLT-C300: A Chinese-language corpus.

The evaluation included both Full-length utterances and Short-length segments (only 1 second of speech) to test robustness under data-constrained conditions. We compared our LSTM-based "r-vectors" against the traditional "i-vector" gold standard.

Key Results and Insights

Performance Gains in LRE

The most striking finding was the improvement in Language Recognition. The SRE component effectively "normalizes" the signal, allowing the LRE component to focus purely on linguistic patterns. In full-length tests, the number of identification errors (IDE) dropped from 29 in the baseline to nearly zero in several collaborative configurations.

Comparison of Identification Rate (IDR) and Identification Error (IDE) across different systems and test lengths.

Robustness on Short Utterances

As seen in the results table, while i-vector models perform well on long audio, the LSTM-based "r-vectors" are significantly more effective for very short (1-second) clips. For example, in the short-length test, the r-vector/Softmax configuration achieved an IDE of 129, compared to 2270 for the i-vector/Cosine baseline.

Gate Sensitivity

The research also explored which parts of the LSTM benefited most from cross-task feedback. Results indicated that the output gate was the most effective location for feeding information into the SRE task, while the forget gate was more beneficial for the LRE task.

Conclusion

Collaborative learning represents a significant step toward more human-like machine audition. By sharing knowledge "on-line" during processing, the model achieves superior performance, particularly in language recognition and short-duration speaker identification. Future work will focus on scaling this framework to more diverse datasets and a larger number of languages.

Summary updated

Abstract

This paper presents a unified model to perform language and speaker recognition simultaneously and altogether. The model is based on a multi-task recurrent neural network where the output of one task is fed as the input of the other, leading to a collaborative learning framework that can improve both language and speaker recognition by borrowing information from each other. Our experiments demonstrated that the multi-task model outperforms the task-specific models on both tasks.
BibTeX
@misc{li2016collaborative,
  author = {Lantian Li and Zhiyuan Tang and Dong Wang and Yang Feng and Shiyue Zhang},
  title = {{Collaborative Learning for Language and Speaker Recognition}},
  year = {2016},
  eprint = {1609.08442},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1609.08442},
}
Figure