TL;DR
This paper introduces "supervised attention" for sequence-to-sequence speech recognition models. By explicitly training the attention mechanism to align with known phonetic boundaries, they achieve faster convergence and significantly improve recognition accuracy, challenging the notion that attention must be inherently interpretable to be effective.
What Problem Does This Paper Address?
Sequence-to-sequence (seq2seq) models are the backbone of modern automatic speech recognition (ASR). A crucial component of these models is the "attention mechanism," which learns to align input acoustic features with output tokens (like phones or characters). The intuition is that attention weights should mirror these alignments, indicating which part of the audio corresponds to which output. However, in standard end-to-end training, attention weights can become "unfaithful" – they might not accurately reflect the true alignments, even if the model performs well. This raises questions about the interpretability of attention and whether its alignment with actual phonetic boundaries is essential for good performance. The authors investigate this by treating the correspondence between attention and alignments as a learning problem itself.
Methods
The core contribution is the introduction of a supervised attention loss ($L_{attn}$). This loss measures the discrepancy between the model's learned attention weights ($a$) and a "target" alignment ($a^*$). The target alignment is derived from pre-computed forced alignments, which provide precise start and end times for phonetic segments in the audio.
The paper explores several ways to convert these forced alignments into target attention weights:
- Uniform weights within segments ($uni\text{-}attn$): Assigns equal weight to all frames within a phonetic segment.
- Point mass at first/center/last frame ($f\text{-}attn, c\text{-}attn, l\text{-}attn$): Places all weight on a single frame within the segment.
- Evenly divided duration ($even\text{-}attn$): Divides the total audio duration evenly among all output tokens, regardless of actual phonetic boundaries.
The supervised attention loss is added to the standard cross-entropy loss ($L_{ce}$), creating a combined loss function: $L = L_{ce} + \gamma L_{attn}$, where $\gamma$ is a hyperparameter. They also explore curriculum learning, where supervised attention is applied only during the initial stages of training and then gradually phased out.
Datasets
The experiments were conducted on the Wall Street Journal (WSJ) dataset. The standard si284 training set was used, split into training and development sets (9:1 ratio). Forced alignments were produced using a speaker-adaptive GMM-HMM system. The models were evaluated on the dev93 and eval92 sets.
Key Insights and Ideas
- Supervised attention significantly improves performance: Explicitly training attention to align with known phonetic boundaries leads to substantial gains in ASR accuracy, even outperforming a strong baseline with dropout.
- Faster convergence: Models trained with supervised attention converge faster in terms of cross-entropy loss, suggesting that this supervision eases the learning process.
- Interpretability is not a prerequisite for performance: The paper demonstrates that even when using "incorrect" alignments (like the $even\text{-}attn$ method, which doesn't reflect true phonetic durations), supervised attention still provides performance benefits. This challenges the common belief that attention must be interpretable or perfectly aligned to be effective.
- Curriculum learning enhances benefits: Phasing out supervised attention after the initial training stages further boosts performance, suggesting that while initial alignment supervision is crucial, it might be too strong a constraint later in training.
- Peaky attention is not necessarily bad: The authors successfully train models with "peaky" attention weights (where only a few frames are attended to) that perform well, refuting another common assumption.
Key Results
- Adding supervised attention (specifically $uni\text{-}attn$) to a strong seq2seq baseline with dropout reduced the phone error rate (PER) from 15.3% to 10.4% on dev93 and from 11.8% to 7.7% on eval92.
- Curriculum training with $uni\text{-}attn$ further reduced PER to 9.5% on dev93 and 7.3% on eval92.
- Even the $even\text{-}attn$ method, which uses an incorrect alignment strategy, showed improvements over the baseline when combined with supervised attention and curriculum learning.
- The paper also shows that supervised attention is complementary to CTC (Connectionist Temporal Classification) loss, further reducing PER when used in a joint training setup.
Limitations and Future Work
- Reliance on forced alignments: The primary limitation is the requirement for accurate forced alignments, which are themselves generated by a separate, potentially complex, system. The authors acknowledge that obtaining these alignments is simpler than ASR itself but still a prerequisite.
- Generalization beyond phonetics: The current work focuses on phonetic recognition. Extending supervised attention to align at higher levels (characters, words, syllables) is a non-trivial but promising future direction, as boundaries at these levels are often more consistent.
- Deeper analysis of attention: Further research is needed to understand why certain attention patterns are attended to and others are not, especially in the case of "peaky" attention.
Summary updated