Abstract
E-Branchformer evolves the Branchformer architecture by introducing an "Enhanced Merging" module that utilizes depth-wise convolution to optimize the fusion of local and global features. By integrating this improved merging mechanism with interleaved feed-forward networks (FFNs), the model achieves a new state-of-the-art (SOTA) Word Error Rate (WER) of 1.81% on the LibriSpeech test-clean benchmark without the use of external training data.
Introduction
In Automatic Speech Recognition (ASR), the Conformer architecture has established itself as a standard by sequentially stacking convolution and self-attention modules to capture local patterns and global context, respectively. Recently, the Branchformer architecture introduced a parallel alternative, processing local and global information in dedicated branches before merging them. However, the original Branchformer relied on a simple linear projection for merging, which treats fusion as a point-wise operation. This approach neglects the temporal relationships between adjacent frames during the critical integration of disparate feature types. E-Branchformer addresses this limitation by refining the merging process to better exploit spatial and temporal dependencies.
Architectural Improvements
The E-Branchformer introduces two primary enhancements to the standard Branchformer block:
1. Enhanced Merge Module
Instead of a simple concatenation and projection, E-Branchformer incorporates a depth-wise convolution layer into the merging process. This allows the model to perform "spatial refinement," ensuring that local and global features are aligned and smoothed across time. The authors explored several variants, including the addition of Squeeze-and-Excitation (SE) blocks for channel-wise gating.
2. Interleaved Feed-Forward Networks (FFNs)
While the original Branchformer omitted standard FFN blocks to prioritize efficiency, E-Branchformer re-introduces them in a "macaron-style" configuration, sandwiching the main branchformer block. This helps refine point-wise information after complex temporal features have been aggregated.
Experimental Setup
The model was evaluated on the LibriSpeech dataset (~1,000 hours of read English speech). Performance was measured on the test-clean and test-other sets. Notably, the study focused on the "no external data" regime, excluding massive unlabelled datasets like Libri-Light for pre-training, which highlights the architectural efficiency of the proposed model.
Results
E-Branchformer sets a new benchmark for Attention-based Encoder-Decoder (AED) models on LibriSpeech.
Key findings include:
- SOTA Performance: The E-Branchformer (Large) model achieved 1.81% WER on
test-cleanand 3.65% WER ontest-otherwhen combined with Internal Language Model Estimation (ILME). - Efficiency: With approximately 149M parameters, the Large model outperforms significantly larger Conformer baselines.
- Ablation Success: Incorporating depth-wise convolution into the merge module alone yielded a 0.1% to 0.3% absolute WER improvement over the standard Branchformer.
Key Insights
The success of E-Branchformer underscores that merging parallel branches is essentially a spatial task. By using a convolution (kernel size 31) in the merge module, the model effectively aligns features across the temporal dimension. Furthermore, the "macaron-style" stacking of FFNs proved more effective than simply increasing the number of Branchformer blocks, suggesting that point-wise refinement remains crucial even in complex parallel architectures.
Conclusion
E-Branchformer demonstrates that sophisticated merging strategies can significantly enhance the performance of parallel-branch architectures in ASR. While the addition of FFN layers and larger kernels increases complexity, the resulting gains in accuracy set a new standard for the field. Future work will explore the application of E-Branchformer to self-supervised learning and other speech-related tasks.
Summary updated