TL;DR
This paper tackles the issue of overconfident predictions in modern Self-Supervised Learning (SSL) based Automatic Speech Recognition (ASR) models, which can hinder the performance of standard beam search decoding. They propose a novel "confidence relaxation" method by aggregating intermediate transformer layer outputs, which improves accuracy and significantly reduces computational cost, especially in low-resource scenarios.
What Problem Does This Paper Address?
Modern ASR systems, particularly those leveraging Self-Supervised Learning (SSL) like Wav2vec 2.0 and HuBERT, have achieved remarkable performance. These models often use a transformer architecture where predictions are made based on the output of the final layers. However, the authors observe that these top layers tend to produce highly confident, often "peaked" probability distributions. This overconfidence can be detrimental to beam search decoding, a common technique that explores multiple potential transcriptions. When a model is too sure about a single token, beam search might miss better, albeit less confident, alternative paths, leading to suboptimal transcriptions. This problem is exacerbated in low-resource settings where models might not have learned the full nuances of speech.
Methods
The core of the proposed method is confidence relaxation through layer aggregation. Instead of relying solely on the output (logits) of the top transformer layer for decoding, the authors propose to aggregate the logits from the top M transformer layers. This aggregation is done by first normalizing the logits from each layer (to prevent higher layers from dominating) and then summing them. This aggregated output is then combined with the original top-layer logits using a learned trade-off coefficient (β). This process effectively "relaxes" the confidence by incorporating information from intermediate layers, which are found to be less confident and more diverse. Crucially, this method requires no additional training or new parameters; it's applied post-fine-tuning.
Datasets
The experiments were conducted using the Librispeech dataset, a widely used benchmark for ASR. They utilized various subsets for fine-tuning, including:
- 10m, 1h, 10h, 100h, 360h, and 960h of labeled training data.
- 'test-clean', 'test-other', 'dev-clean', and 'dev-other' splits for evaluation.
The pre-training was done on either Librispeech (LS-960) or Libri-Light (LL-60K).
Key Insights and Ideas
- Overconfidence is a feature, not a bug (for decoding): SSL ASR models, especially at their top layers, exhibit a strong tendency towards highly confident predictions. This is visualized by the increasing confidence across transformer layers.
- Intermediate layers hold valuable diversity: While top layers are confident, intermediate layers often contain more diverse and less peaked probability distributions, which are crucial for exploring alternative decoding paths.
- Layer aggregation as a "confidence relaxer": By combining information from multiple layers, the proposed method effectively smooths out the overly confident predictions, allowing beam search to consider a wider range of hypotheses.
- Low-resource benefit: The method shows a disproportionately larger benefit in low-resource settings, suggesting that intermediate layers are particularly important when the model hasn't fully converged on top-layer representations.
- Computational efficiency: The aggregation method allows for comparable or better performance with significantly smaller beam widths, leading to substantial reductions in inference time.
Key Results
- Improved WER/CER: The proposed layer aggregation consistently matches or improves Word Error Rate (WER) and Character Error Rate (CER) across various models and dataset sizes compared to the baseline (top-layer only) approach.
- Significant gains in low-resource settings: For datasets with 10 hours or less of labeled data, layer aggregation provides particularly strong improvements.
- Reduced computational cost: The aggregation method enables achieving similar performance with much smaller beam widths. For instance, it matched baseline performance with a beam width of 400 that the baseline required a width of 1500 to achieve on the 'dev-other' set.
- Comparison with Temperature Scaling: Layer aggregation outperformed temperature scaling (another confidence relaxation technique) in most examined cases, highlighting the benefit of leveraging intermediate layer information.
Limitations and Future Work
- Hyperparameter tuning: The optimal number of aggregated layers (M) and the aggregation trade-off coefficient (β) require tuning, which depends on the model size and the amount of labeled data.
- Future Work: The authors suggest further exploring the optimal choices for M and β, and investigating how this approach can be extended to other ASR architectures or even other sequence-to-sequence tasks. They also note that the method is not tied to a specific Language Model (LM) and can be used with different LMs.
Summary updated