Self-Teaching Networks

Liang Lu, Eric Sun, Yifan Gong

TL;DR

This paper introduces "self-teaching networks," a novel training method that improves deep neural network generalization by passing supervision signals from the output layer down to lower layers. This approach helps mitigate the vanishing gradient problem and acts as a regularizer, leading to consistent improvements in speech recognition tasks, outperforming existing methods like label smoothing and confidence penalization.

What Problem Does This Paper Address?

Deep neural networks, despite their success, struggle with generalization, especially to scenarios not well-represented in training data. This is particularly challenging in real-world applications like speech recognition, where acoustic conditions and dialects vary widely. A key technical hurdle is the vanishing gradient problem, which makes it difficult to train very deep networks effectively. While techniques like residual and highway networks have addressed this by creating upward skip connections, this paper explores a complementary approach: passing supervision signals downward from the network's output layer to its lower layers.

Methods

The core innovation is the "self-teaching" mechanism. During training, an auxiliary loss function is introduced. This loss encourages the lower layers of the network to mimic the output layer's behavior by minimizing the divergence (specifically, KL divergence) between their respective posterior probability distributions. This creates a "teacher-student" dynamic within the same network, where the top layer acts as the teacher and lower layers as students. This downward flow of supervision signals serves two purposes: it facilitates gradient propagation to deeper layers and acts as a regularizer, enhancing generalization. The paper focuses on frame-level training for acoustic models using LSTMs.

Datasets

The models were trained on 30 thousand hours of anonymized and transcribed Microsoft production data. Evaluation was performed on four test sets: Cortana (close-talk and far-field speech), DMA-I (in-domain distant microphone array), and DMA-O (out-domain distant microphone array). The DMA-O set is particularly important as it was not covered in the training data, serving as a critical testbed for generalization.

Key Insights and Ideas

  • Downward Supervision Flow: The paper demonstrates that passing supervision signals downward can be as effective, if not more so, than upward skip connections for improving deep network training and generalization.
  • Internal Teacher-Student Learning: The concept of using different layers of the same network as teacher and student is a novel framing for regularization.
  • Regularization Power: The auxiliary loss acts as a powerful regularizer, especially for larger models, suggesting that the benefits of self-teaching become more pronounced as network depth and complexity increase.
  • Entropy Term Importance: The cross-entropy term between the lower and upper layer posteriors appears to be more critical for performance gains than the entropy term itself, which is a key differentiator from confidence penalization.

Key Results

  • Self-teaching networks consistently improved Word Error Rates (WER) across all four evaluation sets compared to baseline models.
  • For smaller LSTM models (31 million parameters), self-teaching achieved improvements over label smoothing and confidence penalization.
  • For larger LSTM models (63 million parameters), self-teaching networks showed even more significant WER reductions, ranging from 3.8% to 8% depending on the test set.
  • The self-teaching approach demonstrated strong generalization capabilities, particularly on the out-of-domain DMA-O test set.

Limitations and Future Work

  • Sequence-Level Training: The current work focuses on frame-level training. Future work will explore sequence-level training, which involves more complex posterior probability computations over lattices.
  • Gradient Vanishing for Very Deep Networks: While the method shows promise, its full potential for training extremely deep recurrent networks needs further evaluation.
  • Resource-Constrained Scenarios: The paper suggests that self-teaching might be particularly beneficial in low-resource conditions or for less common languages, which warrants further investigation.
  • Broader Applications: The authors propose exploring self-teaching for other tasks, such as speaker recognition.

Summary updated

Abstract

We propose self-teaching networks to improve the generalization capacity of deep neural networks. The idea is to generate soft supervision labels using the output layer for training the lower layers of the network. During the network training, we seek an auxiliary loss that drives the lower layer to mimic the behavior of the output layer. The connection between the two network layers through the auxiliary loss can help the gradient flow, which works similar to the residual networks. Furthermore, the auxiliary loss also works as a regularizer, which improves the generalization capacity of the network. We evaluated the self-teaching network with deep recurrent neural networks on speech recognition tasks, where we trained the acoustic model using 30 thousand hours of data. We tested the acoustic model using data collected from 4 scenarios. We show that the self-teaching network can achieve consistent improvements and outperform existing methods such as label smoothing and confidence penalization.
BibTeX
@inproceedings{lu2019self,
  author = {Liang Lu and Eric Sun and Yifan Gong},
  title = {{Self-Teaching Networks}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure