Self-Teaching Networks

Liang Lu, Eric Sun, Yifan Gong

Abstract

We propose self-teaching networks to improve the generalization capacity of deep neural networks. The idea is to generate soft supervision labels using the output layer for training the lower layers of the network. During the network training, we seek an auxiliary loss that drives the lower layer to mimic the behavior of the output layer. The connection between the two network layers through the auxiliary loss can help the gradient flow, which works similar to the residual networks. Furthermore, the auxiliary loss also works as a regularizer, which improves the generalization capacity of the network. We evaluated the self-teaching network with deep recurrent neural networks on speech recognition tasks, where we trained the acoustic model using 30 thousand hours of data. We tested the acoustic model using data collected from 4 scenarios. We show that the self-teaching network can achieve consistent improvements and outperform existing methods such as label smoothing and confidence penalization.

Introduction

As neural networks grow deeper, they face two primary hurdles: the vanishing gradient problem and generalization. In deep LSTMs (Long Short-Term Memory networks), gradients often become "noisy" or dissipate before they can effectively update the earliest layers. Furthermore, while models trained on massive datasets (like 30,000 hours of speech) are powerful, they often overfit to the dominant domains in the training data, failing to generalize to "out-of-domain" scenarios—such as a speech recognizer trained on close-talk audio struggling with distant microphones in a noisy room.

Traditional fixes like Residual (skip) connections or standard regularization (Dropout, L1/L2) often see diminishing returns as datasets scale into the tens of thousands of hours. We propose a way to provide higher-quality guidance to the bottom of the network without adding computational overhead during inference.

Methodology: The Self-Teaching Mechanism

The core innovation is the Self-Teaching mechanism. In a standard setup, only the final layer receives the "ground truth" signal. In a Self-Teaching Network, the top layer acts as a mentor to internal layers.

Architecture of a Self-Teaching Network showing the auxiliary KL loss between the top posterior and a lower layer posterior.

As shown in Figure 1, the process involves:

  1. Internal Supervision: An auxiliary softmax layer is attached to a lower layer (e.g., the 3rd layer of a 6-layer LSTM).
  2. The "Teacher" within: The top layer acts as a teacher, generating "soft labels" (a probability distribution across all possible classes).
  3. Auxiliary Loss: The model is trained using a composite loss function. It minimizes the standard cross-entropy (CE) at the top layer plus the Kullback-Leibler (KL) divergence between the top layer’s distribution and the lower layer’s distribution.
  4. Gradient Shortcut: This auxiliary loss acts as a "skip connection" for the supervision signal, passing it directly from the top to the middle of the network.

Unlike traditional Teacher-Student distillation, where a large pre-trained model teaches a smaller one, here the "teacher" and "student" are different layers of the same model being updated simultaneously.

Experimental Setup

The approach was evaluated using a massive industrial-scale dataset consisting of 30,000 hours of anonymized Microsoft production speech data, covering both close-talk and far-field conditions. To test robustness, we used four distinct evaluation scenarios:

  • Cortana: Mixed close-talk and far-field speech.
  • DMA-I (In-domain): Distant microphone array data similar to the training set.
  • DMA-O (Out-domain): Distant microphone array data not represented in the training set—the ultimate test for generalization.

Key Results

We compared Self-Teaching (ST) against strong baselines like Label Smoothing (LS) and Confidence Penalization (CP) using Word Error Rate (WER).

Word Error Rate (WER) comparison across different systems and domains.

As seen in Figure 2, Self-Teaching provided consistent improvements across all test sets. Notably, on the difficult DMA-O (out-of-domain) task, the self-teaching model (ST with H) achieved a WER of 19.13%, outperforming the baseline (19.56%), LS (19.35%), and CP (19.44%).

Further experiments on a larger 63-million parameter model showed even more significant gains. On the DMA-O task, the larger self-teaching model reduced WER from 17.92% to 17.32%. Because the auxiliary layers are discarded after training, the resulting model maintains the same inference speed as a standard LSTM.

Conclusion

Self-teaching networks offer a powerful, data-driven regularization method. By using the model’s own evolving logic to soften targets, it provides a richer signal that prevents overconfidence and improves feature extraction in lower layers. The method is particularly effective as model size increases, providing gains comparable to doubling model capacity without the associated inference costs. Future work will explore extending this to sequence-level training and other architectures like Transformers.

Summary updated

Abstract

We propose self-teaching networks to improve the generalization capacity of deep neural networks. The idea is to generate soft supervision labels using the output layer for training the lower layers of the network. During the network training, we seek an auxiliary loss that drives the lower layer to mimic the behavior of the output layer. The connection between the two network layers through the auxiliary loss can help the gradient flow, which works similar to the residual networks. Furthermore, the auxiliary loss also works as a regularizer, which improves the generalization capacity of the network. We evaluated the self-teaching network with deep recurrent neural networks on speech recognition tasks, where we trained the acoustic model using 30 thousand hours of data. We tested the acoustic model using data collected from 4 scenarios. We show that the self-teaching network can achieve consistent improvements and outperform existing methods such as label smoothing and confidence penalization.
BibTeX
@inproceedings{lu2019self,
  author = {Liang Lu and Eric Sun and Yifan Gong},
  title = {{Self-Teaching Networks}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure