Deja-vu: Double Feature Presentation in Deep Transformer Networks

Andros Tjandra, Chunxi Liu, Frank Zhang, Xiaohui Zhang, Yongqiang Wang, Gabriel Synnaeve, Satoshi Nakamura, Geoffrey Zweig

Introduction

In standard deep neural networks for Automatic Speech Recognition (ASR), raw acoustic features, such as Mel-spectrograms, are typically processed only at the input layer. As these features pass through successive layers, they are progressively transformed into increasingly abstract representations. While this abstraction is necessary for high-level phonetic modeling, the authors of this paper argue that original acoustic nuances are often lost by the time data reaches the final layers of a deep network.

This paper introduces a "deja-vu" mechanism for deep Transformer networks. The core idea is to allow the model to "look back" at raw input features halfway through its processing. By re-examining the raw audio in the context of partial hypotheses formed in earlier layers, the network can better adapt to specific noise conditions or resolve phonetic ambiguities that might have been smoothed over during initial abstraction.

Methodology

The proposed architecture builds upon deep Transformer acoustic models (up to 36 layers) and incorporates two primary modifications: Iterated Loss and Feature Re-Presentation.

Iterated Loss

To ensure that intermediate layers develop phonetically meaningful representations early in the network, the authors attach auxiliary model heads (comprising MLPs and softmax layers) to middle stages (e.g., layers 8, 16, and 24). A Connectionist Temporal Classification (CTC) or Cross-Entropy (CE) loss is applied at these points, providing stronger gradient signals and forcing the network to resolve ambiguities sooner.

Feature Re-Presentation (The "Deja-Vu" Mechanism)

At a designated intermediate layer, the original input features are re-injected into the network. To merge these raw features with current intermediate activations, both are projected to a common dimension and concatenated along the time axis, effectively doubling the sequence length. A self-attention layer is then employed where the intermediate activations serve as the "Query" to extract relevant information from the raw features, which act as "Keys" and "Values."

The Deja-vu architecture featuring iterated loss and intermediate feature re-presentation.

Experimental Results

The model was evaluated on the Librispeech 960-hour benchmark and an in-house Video Dataset containing 1,000 hours of diverse English speech.

Librispeech Performance

On a 24-layer Transformer trained with CTC, the "deja-vu" method yielded a 10-20% relative improvement in Word Error Rate (WER). When scaled to a 36-layer model and combined with SpecAugment, the system achieved state-of-the-art results of 3.2% on test-clean and 7.2% on test-other.

Performance comparison on Librispeech test sets across different model configurations.

Real-world Generalization

The method proved robust on noisy, real-world video data, showing relative improvements ranging from 3.2% to 13% across curated, clean, and noisy subsets. Ablation studies confirmed that while iterated loss alone provides a significant boost, the addition of feature re-presentation consistently delivers the best overall performance across both CTC and Hybrid (CE) architectures.

Key Insights and Limitations

  • Mid-network supervision is powerful: Forcing early layers to predict the final output improves gradient flow and forces earlier resolution of phonetic ambiguity.
  • Raw data remains relevant: Re-introducing features deep in the network acts as a highly efficient "secondary feature analysis," similar to a two-pass decoding system within a single forward pass.
  • Concatenation Strategy: The authors found that concatenating along the time axis and using self-attention to mix features outperformed simple feature-axis concatenation.

Limitations: The study is primarily empirical, and the doubling of sequence length during the re-presentation step introduces additional computational and memory overhead. Future work may explore dynamic routing to determine which layers benefit most from looking back at the input features.

Summary updated

BibTeX
@inproceedings{tjandra2020deja,
  author = {Andros Tjandra and Chunxi Liu and Frank Zhang and Xiaohui Zhang and Yongqiang Wang and Gabriel Synnaeve and Satoshi Nakamura and Geoffrey Zweig},
  title = {{Deja-vu: Double Feature Presentation in Deep Transformer Networks}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  doi = {10.1109/icassp40776.2020.9052964},
}
Figure