DEJA-VU: Double Feature Presentation and Iterated Loss in Deep Transformer Networks

Andros Tjandra, Chunxi Liu, Frank Zhang, Xiaohui Zhang, Yongqiang Wang, Gabriel Synnaeve, Satoshi Nakamura, Geoffrey Zweig

TL;DR

This paper introduces "DEJA-VU," a novel approach for deep acoustic models that allows them to re-examine input features at multiple stages within a Transformer network. By applying loss functions at intermediate layers and enabling these layers to attend to both original and processed features, the model achieves significant improvements in speech recognition accuracy.

What Problem Does This Paper Address?

Traditional deep neural networks for acoustic modeling process input features sequentially, abstracting them layer by layer. While effective, this can lead to a loss of crucial information or an inability to adapt to varying acoustic conditions or partial hypotheses. Existing adaptation methods often involve complex, multi-pass processes. This paper aims to enable acoustic models to dynamically re-evaluate and refine their understanding of input features within the network, inspired by the idea of "re-seeing" or "deja vu" for the model.

Methods

The core innovation lies in two main components:

  • Iterated Loss: Instead of applying a loss function only at the final output layer, this method introduces auxiliary loss functions at intermediate Transformer layers. This encourages the network to develop meaningful phonetic or graphemic hypotheses earlier in the processing pipeline, providing more informative gradients for training.
  • Double Feature Presentation (Feature Re-Presentation): At intermediate layers, the model is equipped with an additional mechanism that allows it to attend to both the original input features and the activations from the previous layer. This is achieved by projecting both feature streams, concatenating them along the time axis, and then using a self-attention mechanism to extract relevant information. This allows the model to "re-examine" its input in light of what it has already processed.

The architecture uses standard Transformer modules, but the key novelty is the integration of these auxiliary losses and the feature re-presentation mechanism.

Datasets

The proposed methods were evaluated on two datasets:

  • Librispeech: A widely used dataset for English speech recognition, comprising 960 hours of training data.
  • Video Dataset: A large-scale collection of public and anonymized English videos, including 1000 hours of training data.

Key Insights and Ideas

  • Intermediate supervision is powerful: Applying loss functions at intermediate layers significantly improves performance on its own, suggesting that guiding the network's learning process at multiple stages is more effective than a single final loss.
  • Re-examining features is beneficial: Allowing intermediate layers to access and re-process original input features, in addition to their own activations, leads to further performance gains. This suggests that models can benefit from a form of "memory" or "context" that spans across different levels of abstraction.
  • Feature concatenation strategy matters: The paper found that concatenating features along the time axis (as opposed to the feature axis) for the Key and Value in the attention mechanism yielded better results.
  • Transformer's adaptability: The Transformer architecture, with its self-attention mechanism, is well-suited for integrating these new components, allowing for efficient parallelization and effective feature aggregation.

Key Results

The DEJA-VU approach demonstrated significant improvements across both datasets:

  • Librispeech: Relative improvements of 10-20% were observed compared to the baseline Transformer models.
  • Video Dataset: Relative improvements of 3.2-13% were achieved.

These gains were consistent across both CTC (Connectionist Temporal Classification) and hybrid ASR (Automatic Speech Recognition) systems. The benefits were seen both from the iterated loss alone and from the combined iterated loss and feature re-presentation.

Limitations and Future Work

  • Computational Cost: While Transformers offer parallelization benefits, the addition of intermediate losses and feature re-presentation mechanisms can increase computational complexity and training time. The paper notes that the baseline with 36 layers performed similarly to a 24-layer baseline, but the proposed methods improved the 36-layer model, indicating that deeper models can benefit more, but at a higher cost.
  • Hyperparameter Tuning: The effectiveness of the auxiliary loss coefficient ($\lambda$) and the placement of intermediate layers were determined through preliminary experiments, suggesting that optimal settings might require careful tuning.
  • Future Work: The paper implicitly suggests exploring different placements of feature re-presentation and iterated loss, as well as investigating more sophisticated ways to combine features from different layers. Further research could also explore the interpretability of these intermediate representations.

Summary updated

BibTeX
@inproceedings{tjandra2020deja,
  author = {Andros Tjandra and Chunxi Liu and Frank Zhang and Xiaohui Zhang and Yongqiang Wang and Gabriel Synnaeve and Satoshi Nakamura and Geoffrey Zweig},
  title = {{DEJA-VU: Double Feature Presentation and Iterated Loss in Deep Transformer Networks}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  doi = {10.1109/icassp40776.2020.9052964},
}
Figure