Speaker-Invariant Feature-Mapping for Distant Speech Recognition via Adversarial Teacher-Student Learning

Long Wu, Hangting Chen, Li Wang, Pengyuan Zhang, Yonghong Yan

Overview

Distant Speech Recognition (DSR) remains a significant challenge in the field of audio processing due to environmental factors like background noise, room reverberation, and overlapping speech. Researchers have introduced SIAFM-TS, a unified framework designed to enhance noisy audio while simultaneously stripping away speaker-specific variability. By combining adversarial training with teacher-student learning, the system achieves a substantial reduction in Word Error Rate (WER) on the AMI meeting dataset.

The Challenge of Distant Speech

Capturing clear speech from a distance involves navigating a "messy cocktail" of acoustic interference. Traditional Feature Mapping (FM) techniques attempt to translate these degraded signals into clean, close-talk versions. However, these methods often struggle with inter-speaker variability—the unique spectral characteristics of individual voices. A mapping optimized for one speaker may fail for another. The authors' goal was to create a "speaker-invariant" model that prioritizes linguistic content over speaker identity, ensuring robust performance across diverse users.

Methodology: The SIAFM-TS Framework

The proposed framework integrates three core components to address the complexities of DSR:

  1. Senone-Aware Feature-Mapping (AFM): Rather than relying solely on Mean Squared Error (MSE) to map noisy audio to clean audio, the mapper is jointly trained with an acoustic model. This ensures that the resulting "enhanced" features are specifically optimized for distinguishing speech sounds (senones).
  2. Adversarial Speaker-Invariant Training: To remove speaker-specific traits, a speaker classifier is added to the network. The feature mapper is trained to maximize the classifier's error (adversarial loss), forcing the network to learn representations that are neutral across different speakers.
  3. Teacher-Student (TS) Learning: A "Teacher" model, trained on high-quality Individual Headset Microphone (IHM) data, provides "soft labels" (probability distributions) to the "Student" model. These labels transfer "dark knowledge" about the relationships between speech sounds, helping the student bridge the performance gap between distant and close-talk audio.
The unified SIAFM-TS framework architecture, integrating feature-mapping, adversarial training, and teacher-student distillation.

Experimental Setup

The study utilized the AMI Meeting Corpus, a well-known benchmark for multi-party meeting recognition. The training set included 80 hours of speech from 155 speakers, while the test set comprised 9 hours from 63 unseen speakers. Performance was evaluated by comparing Single Distant Microphone (SDM) data against the high-quality IHM baseline.

Key Insights: Achieving Speaker Invariance

A critical finding of the research is the effectiveness of the adversarial component. In standard models, male and female speakers often occupy distinct clusters in the feature space, which can lead to biased or inconsistent recognition.

t-SNE visualizations of speaker clusters across different models, demonstrating the achievement of speaker-invariant representations in SIAFM.

As shown in the t-SNE visualizations above, while standard SDM and IHM features show clear speaker clustering, the adversarial SIAFM method causes these clusters to overlap significantly. This proves the model successfully learned speaker-invariant representations, focusing on linguistic content rather than individual voice characteristics.

Results and Performance

The SIAFM-TS model demonstrated consistent improvements over baseline methods. Compared to the standard industry approach of Multi-Conditional Training (MCT), SIAFM-TS achieved a 4.2% relative WER reduction on overlapped speech and a 6.3% reduction on non-overlapped speech.

Acoustic model WER(over) SDM-Eval WER(nonover) SDM-Eval
SDM 54.0 44.5
MCT 53.0 43.2
AFM 52.1 42.5
SIAFM 51.4 41.4
SIAFM-TS 50.8 40.5

Qualitatively, spectrogram analysis confirmed that the feature mapping effectively suppresses reverberation and restores speech information that was previously blurred in the distant microphone recordings.

Conclusion and Future Work

While SIAFM-TS significantly narrows the gap, a performance difference still exists between distant and close-talk recognition. Future research may explore integrating these adversarial techniques with multi-channel beamforming (using microphone arrays) and investigating dynamic tuning of loss weights to further optimize the trade-off between enhancement and speaker invariance.

Summary updated

Abstract

Feature mapping (FM) jointly trained with acoustic model (AFM) is commonly used for single-channel speech enhancement. However, the performance is affected by the inter-speaker variability. In this paper, we propose speaker-invariant AFM (SIAFM) aiming at curtailing the inter-talker variability while achieving speech enhancement. In SIAFM, a feature-mapping network, an acoustic model and a speaker classifier network are jointly optimized to minimize the feature-mapping loss and the senone classification loss, and simultaneously min-maximize the speaker classification loss. Evaluated on AMI dataset, the proposed SIAFM achieves 4.8% and 7.0% relative word error rate (WER) reduction on the overlapped and non-overlapped condition over the baseline acoustic model trained with single distant microphone (SDM) data. Additionally, the SIAFM obtains 3.0% relative overlapped WER and 4.2% relative non-overlapped WER decrease over the multi-conditional (MCT) acoustic model. To further promote the performance of SIAFM, we employ teacher-student learning (TS), in which the posterior probabilities generated by the individual headset microphone (IHM) data can be used in lieu of labels to train the SIAFM model. The experiments show that compared with MCT model, SIAFM with TS (SIAFM-TS) can reach 4.2% relative overlapped WER and 6.3% relative non-overlapped WER decrease respectively.
BibTeX
@inproceedings{wu2019speaker,
  author = {Long Wu and Hangting Chen and Li Wang and Pengyuan Zhang and Yonghong Yan},
  title = {{Speaker-Invariant Feature-Mapping for Distant Speech Recognition via Adversarial Teacher-Student Learning}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-2136},
}
Figure