TL;DR
This paper introduces a novel multi-task, multi-resolution end-to-end (E2E) speech recognition architecture that combines character-level and Byte-Pair Encoding (BPE) subword units. By jointly training on both granularities and incorporating a cross-attention mechanism, the model achieves state-of-the-art performance on the LibriSpeech dataset with a significantly smaller footprint than previous top models.
What Problem Does This Paper Address?
End-to-end (E2E) Automatic Speech Recognition (ASR) models, particularly attention-based encoder-decoder (AED) models, have shown great promise due to their ability to implicitly capture language information, reducing the need for external language models and enabling on-device applications. However, a significant challenge for E2E models is their performance degradation on smaller datasets compared to traditional hybrid HMM-DNN systems. While larger subword units like BPE have been shown to improve performance and model size trade-offs, there's a need to further enhance smaller E2E models to match the performance of much larger systems without a proportional increase in parameters. This paper aims to bridge this gap by proposing a more sophisticated E2E architecture that leverages multi-task learning and multi-resolution representations.
Methods
The core of this work is a Multi-Task Multi-Resolution (MTMR) attention-based encoder-decoder (AED) model. The architecture features a shared encoder stack that processes the input audio. This encoder is trained with four different losses: character-level CTC loss, BPE-level CTC loss, character-level cross-entropy (CE) loss, and BPE-level CE loss.
A key innovation is the hierarchical character-to-BPE (C2B) approach. The model utilizes two distinct encoder stacks: a lower one for character sequences and an upper one for BPE sequences, each with different temporal subsampling factors. Crucially, a char-to-BPE cross-attention decoder is introduced. This decoder takes embeddings from both the character and BPE encoder stacks, allowing it to combine information at different linguistic granularities and temporal resolutions. The paper explores two ways of combining these embeddings: Multi-Head Attention (MHA) and Multi-Head Decoder (MHD), with MHD showing superior performance.
The authors also investigate different training strategies, including curriculum learning (training one task at a time) versus multi-task learning (training all tasks simultaneously). They find that a purely multi-task approach, trained from scratch, yields the best results.
Datasets
The experiments were conducted on the LibriSpeech corpus, a widely used benchmark for ASR research. The training set comprises approximately 960 hours of transcribed audio. The dev-clean and dev-other subsets (around 5 hours each) were used for validation, and the test-clean and test-other subsets (also 5 hours each) were used for final performance evaluation. No new benchmarks were introduced.
Key Insights and Ideas
- Jointly learning character and BPE representations is beneficial: The paper demonstrates that training on both character and BPE targets simultaneously acts as a regularizer, improving the learning of both sequence types.
- Multi-resolution information is key: Combining audio features processed at different temporal resolutions (character vs. BPE) allows the model to capture finer details and broader context, leading to better performance.
- Cross-attention between granularities is powerful: The char-to-BPE cross-attention decoder is a critical component, enabling the model to leverage information from both character and subword levels effectively.
- Multi-Head Decoder (MHD) outperforms Multi-Head Attention (MHA) in this context: The specific way of combining information from different encoders and tasks at the decoder level significantly impacts performance.
- Training from scratch with multi-task learning is superior to curriculum learning: For this architecture, a simultaneous multi-task training approach yields better results than a staged, curriculum-based approach.
Key Results
The proposed Multi-Task Multi-Resolution (MTMR) C2B AED model achieved significant improvements:
- On the LibriSpeech corpus, the model achieved a relative Word Error Rate (WER) improvement of approximately 15% compared to a baseline bidirectional full-attention (BFA) model with the same parameter count.
- The best performing model, an 8-layer BFA model with the MTMR C2B MHD decoder, achieved a test-clean WER of 3.45% and a test-other WER of 11.25%.
- This performance is comparable to a significantly larger (approximately 3 times bigger) 1024-unit BFA model on the test-clean set and even better on the test-other set, especially when shallow fusion with an RNN-LM is applied. This highlights the efficiency of the proposed architecture.
Limitations and Future Work
- Limitations: The paper notes that while the proposed model significantly improves performance for smaller footprint models, it doesn't fully match the performance of the largest state-of-the-art models without external LMs. The motivation for including a separate character-based attention decoder, beyond improving CTC encoder performance, could be further explored, particularly in the context of joint beam-search decoders.
- Future Work: The authors suggest exploring the design of a joint beam-search decoder that utilizes both character and BPE attention decoders. They also mention that the proposed architecture is generalizable and can be extended to combine more tasks and resolutions.
Summary updated