Conditional Teacher-Student Learning

Zhong Meng, Jinyu Li, Yong Zhao, Yifan Gong

Abstract

Teacher-student (T/S) learning is effective for domain adaptation and model compression, but teacher models occasionally produce incorrect guidance that misleads the student. We propose a conditional T/S learning scheme where a "smart" student model selectively learns from either the teacher's posterior probabilities or ground truth labels, depending on the teacher's correctness. Unlike linear interpolation, this conditional approach exclusively uses the teacher's guidance when it is correct and reverts to ground truth otherwise. Evaluated on the CHiME-3 and Microsoft short message dictation datasets, this method achieved 9.8% and 12.8% relative word error rate (WER) reductions, respectively, over standard T/S and speaker-independent baselines.

Introduction

Teacher-Student (T/S) learning, also known as knowledge distillation, has become a cornerstone of modern machine learning for tasks like model compression and domain adaptation. In this framework, a "student" model is trained to mimic the output probabilities (soft labels) of a more complex "teacher" model. These soft labels are highly informative because they capture "dark knowledge"—the subtle relationships between different output classes that are lost in one-hot ground truth labels.

However, a fundamental flaw in standard T/S learning is the assumption of an infallible teacher. When a teacher model makes a mistake, its soft labels can provide misleading signals, steering the student toward suboptimal performance. Existing workarounds, such as "interpolated" T/S learning, attempt to balance teacher guidance with ground truth labels using a weighted average. The authors of this paper argue that such interpolation is problematic: it destroys the nuanced probabilistic structure of the teacher's output and introduces a sensitive hyperparameter ($\lambda$) that requires extensive tuning.

Conditional Teacher-Student Learning

To overcome these limitations, the researchers propose Conditional T/S Learning. This approach treats the student as an active learner that evaluates the teacher's reliability on a frame-by-frame basis.

Architecture of the Conditional Teacher-Student Learning framework.

As illustrated in the diagram above, the training process incorporates a conditional switch. For every input sample, the system determines if the teacher's top prediction matches the ground truth label:

  • If the teacher is correct: The student uses the teacher’s full posterior distribution as its target. This preserves the rich structural information when the teacher is trustworthy.
  • If the teacher is incorrect: The student ignores the teacher's output and instead uses the one-hot ground truth hard label.

This "best of both worlds" strategy ensures that the student benefits from the teacher's dark knowledge without being corrupted by its errors. Crucially, it removes the need for the interpolation weight $\lambda$, simplifying the training pipeline.

Experimental Results

The proposed method was tested on two major benchmarks: the CHiME-3 dataset for domain adaptation in noisy environments and a Microsoft internal dataset for speaker adaptation.

Domain Adaptation (CHiME-3)

The CHiME-3 task involves recognizing speech in challenging environments like buses (BUS), cafes (CAF), pedestrian areas (PED), and streets (STR). The Conditional T/S method was compared against unadapted models, hard label training, and various configurations of soft and interpolated T/S learning.

Word Error Rate (WER) results on the CHiME-3 dataset for various adaptation methods.

As shown in the table above, Conditional T/S learning consistently outperformed all other methods across all noise conditions. It achieved an average Word Error Rate (WER) of 16.42%, a 9.8% relative reduction over the standard soft T/S approach (18.20% WER).

Speaker Adaptation

In speaker adaptation tasks, the goal is to tune a speaker-independent model to a specific user's voice. In supervised settings, the method achieved a 12.17% WER, representing a 12.8% relative improvement over the speaker-independent baseline. Even in unsupervised adaptation, where ground truth labels are replaced by model hypotheses, the conditional approach provided a 5.3% relative gain, demonstrating that the method remains effective even when the "ground truth" is a high-confidence model hypothesis.

Conclusion

Conditional T/S learning represents a significant step forward in knowledge distillation. By making the student "smart" enough to know when to trust its teacher, the framework avoids the pitfalls of error propagation and the complexities of hyperparameter tuning. The results across domain and speaker adaptation tasks confirm that conditional switching is a more robust and effective strategy than traditional linear interpolation for transferring knowledge between models.

Summary updated

Abstract

The teacher-student (T/S) learning has been shown to be effective for a variety of problems such as domain adaptation and model compression. One shortcoming of the T/S learning is that a teacher model, not always perfect, sporadically produces wrong guidance in form of posterior probabilities that misleads the student model towards a suboptimal performance. To overcome this problem, we propose a conditional T/S learning scheme, in which a "smart" student model selectively chooses to learn from either the teacher model or the ground truth labels conditioned on whether the teacher can correctly predict the ground truth. Unlike a naive linear combination of the two knowledge sources, the conditional learning is exclusively engaged with the teacher model when the teacher model's prediction is correct, and otherwise backs off to the ground truth. Thus, the student model is able to learn effectively from the teacher and even potentially surpass the teacher. We examine the proposed learning scheme on two tasks: domain adaptation on CHiME-3 dataset and speaker adaptation on Microsoft short message dictation dataset. The proposed method achieves 9.8% and 12.8% relative word error rate reductions, respectively, over T/S learning for environment adaptation and speaker-independent model for speaker adaptation.
BibTeX
@inproceedings{meng2019conditional,
  author = {Zhong Meng and Jinyu Li and Yong Zhao and Yifan Gong},
  title = {{Conditional Teacher-Student Learning}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  doi = {10.1109/icassp.2019.8683438},
}
Figure