TL;DR
This paper introduces a novel approach to automatic speech recognition (ASR) by jointly training end-to-end (E2E) and DNN-HMM hybrid systems. By sharing components between these two traditionally separate architectures, the system effectively transfers knowledge, leading to improved ASR accuracy compared to either system alone. This work bridges the gap between the global optimization of E2E models and the stable frame-by-frame processing of DNN-HMM systems.
What Problem Does This Paper Address?
Automatic Speech Recognition (ASR) systems have seen significant advancements with deep learning. Two dominant paradigms have emerged: DNN-HMM hybrid systems and end-to-end (E2E) systems. DNN-HMM systems, while effective, rely on a complex pipeline of acoustic models, pronunciation models, and language models, which can lead to a lack of global optimization. Conversely, E2E systems, such as those based on Connectionist Temporal Classification (CTC) or encoder-decoder architectures, aim for direct speech-to-text conversion and can achieve global optimization. However, E2E systems can sometimes suffer from issues like redundant predictions or the vanishing of important symbols.
Previous attempts to combine these systems often involved using one to rescore the output of the other, treating them as independent entities. This paper addresses the limitation of this independent training by proposing a joint system where components are shared. The core problem is how to effectively leverage the complementary strengths of both E2E and DNN-HMM systems by enabling them to learn from each other's internal representations, rather than just their final outputs.
Methods
The paper proposes a joint ASR system that integrates an E2E encoder-decoder model with a DNN-HMM hybrid system through a shared network. The key innovation lies in using the internal outputs of the DNN acoustic model within the DNN-HMM system as auxiliary features for the E2E system.
Specifically:
- DNN-HMM Hybrid System: This system uses a DNN to estimate frame-level acoustic features, which are then fed into a Hidden Markov Model (HMM) for phoneme state estimation. The internal outputs of this DNN (e.g., from a layer before the final softmax) are considered rich in information about phoneme discrimination.
- End-to-End (E2E) System: This is an encoder-decoder model that directly maps acoustic features to a sequence of symbols (characters or words).
- Shared Network: The proposed system feeds both the standard acoustic features (e.g., FBANK) and the internal outputs from the DNN acoustic model into the encoder of the E2E system. This allows the E2E model to benefit from the phoneme-level discriminative information learned by the DNN-HMM's acoustic model.
- Joint Training: While the paper mentions parameters are optimized separately, the sharing of internal representations during inference and potentially during training (though not explicitly detailed as joint end-to-end training) is the core of the method. The final ASR score is a combination of scores from both systems.
Datasets
The experiments were conducted on the Corpus of Spontaneous Japanese (CSJ). This corpus is described as a Japanese lecture corpus. The paper provides details on the training, development, and evaluation splits of this dataset, including the number of characters, words, and hours for each. No new benchmarks were introduced; existing datasets were used for evaluation.
Key Insights and Ideas
- Complementary Strengths: The paper strongly suggests that E2E and DNN-HMM systems have distinct advantages. E2E models excel at global optimization, while DNN-HMM models offer stable frame-by-frame processing.
- Value of Internal Representations: The internal outputs of a DNN acoustic model are not just intermediate steps but contain valuable, discriminative information about phonemes that can significantly benefit an E2E system.
- Knowledge Transfer: By sharing these internal representations, knowledge is effectively transferred from the DNN-HMM acoustic model to the E2E model, leading to improved performance.
- Synergy: Combining acoustic features with internal DNN outputs for the E2E system yields better results than using either feature source alone.
Key Results
The experimental results on the CSJ dataset demonstrate the effectiveness of the proposed joint system:
- The joint system that combined FBANK features with internal outputs of the acoustic model achieved the lowest Character Error Rate (CER) of 9.18%.
- This joint system outperformed the standalone E2E system (12.95% CER with FBANK) and the standalone DNN-HMM system (10.85% CER).
- Even when using only the internal outputs of the acoustic model as input to the E2E system, the joint system achieved a CER of 9.35%, which was better than a DNN-HMM system augmented with an LSTM-RNNLM (10.17% CER).
- Figure 2 shows that using internal outputs for training the E2E model leads to higher initial prediction accuracies and better overall performance during training compared to using only FBANK features.
Limitations and Future Work
- Limitations: The paper notes that in their previous work, the DNN-HMM and E2E systems were independently trained. While this paper proposes sharing internal representations, the current experimental setup still involves separate optimization of parameters for the DNN acoustic model and the E2E model. The paper implies that the "joint system" refers to the combination of their outputs and shared feature inputs, rather than a fully end-to-end jointly trained model.
- Future Work: The authors explicitly suggest joint training of the DNN-HMM hybrid ASR system and the E2E ASR system as a promising direction for further improving ASR performance. This would involve optimizing all parameters simultaneously to maximize the benefit of the shared network.
Summary updated