E-Branchformer: Branchformer with Enhanced Merging for Speech Recognition

Kwangyoun Kim, Felix Wu, Yifan Peng, Jing Pan, Prashant Sridhar, Kyu J. Han, Shinji Watanabe

Abstract

E-Branchformer evolves the Branchformer architecture by introducing an "Enhanced Merging" module that utilizes depth-wise convolution to optimize the fusion of local and global features. By integrating this improved merging mechanism with interleaved feed-forward networks (FFNs), the model achieves a new state-of-the-art (SOTA) Word Error Rate (WER) of 1.81% on the LibriSpeech test-clean benchmark without the use of external training data.

Introduction

In Automatic Speech Recognition (ASR), the Conformer architecture has established itself as a standard by sequentially stacking convolution and self-attention modules to capture local patterns and global context, respectively. Recently, the Branchformer architecture introduced a parallel alternative, processing local and global information in dedicated branches before merging them. However, the original Branchformer relied on a simple linear projection for merging, which treats fusion as a point-wise operation. This approach neglects the temporal relationships between adjacent frames during the critical integration of disparate feature types. E-Branchformer addresses this limitation by refining the merging process to better exploit spatial and temporal dependencies.

Architectural Improvements

The E-Branchformer introduces two primary enhancements to the standard Branchformer block:

1. Enhanced Merge Module

Instead of a simple concatenation and projection, E-Branchformer incorporates a depth-wise convolution layer into the merging process. This allows the model to perform "spatial refinement," ensuring that local and global features are aligned and smoothed across time. The authors explored several variants, including the addition of Squeeze-and-Excitation (SE) blocks for channel-wise gating.

Comparison of different merge module architectures, including the proposed enhanced versions with depth-wise convolution and SE blocks.

2. Interleaved Feed-Forward Networks (FFNs)

While the original Branchformer omitted standard FFN blocks to prioritize efficiency, E-Branchformer re-introduces them in a "macaron-style" configuration, sandwiching the main branchformer block. This helps refine point-wise information after complex temporal features have been aggregated.

Experimental Setup

The model was evaluated on the LibriSpeech dataset (~1,000 hours of read English speech). Performance was measured on the test-clean and test-other sets. Notably, the study focused on the "no external data" regime, excluding massive unlabelled datasets like Libri-Light for pre-training, which highlights the architectural efficiency of the proposed model.

Results

E-Branchformer sets a new benchmark for Attention-based Encoder-Decoder (AED) models on LibriSpeech.

Performance comparison of E-Branchformer against various SOTA models on LibriSpeech.

Key findings include:

  • SOTA Performance: The E-Branchformer (Large) model achieved 1.81% WER on test-clean and 3.65% WER on test-other when combined with Internal Language Model Estimation (ILME).
  • Efficiency: With approximately 149M parameters, the Large model outperforms significantly larger Conformer baselines.
  • Ablation Success: Incorporating depth-wise convolution into the merge module alone yielded a 0.1% to 0.3% absolute WER improvement over the standard Branchformer.

Key Insights

The success of E-Branchformer underscores that merging parallel branches is essentially a spatial task. By using a convolution (kernel size 31) in the merge module, the model effectively aligns features across the temporal dimension. Furthermore, the "macaron-style" stacking of FFNs proved more effective than simply increasing the number of Branchformer blocks, suggesting that point-wise refinement remains crucial even in complex parallel architectures.

Conclusion

E-Branchformer demonstrates that sophisticated merging strategies can significantly enhance the performance of parallel-branch architectures in ASR. While the addition of FFN layers and larger kernels increases complexity, the resulting gains in accuracy set a new standard for the field. Future work will explore the application of E-Branchformer to self-supervised learning and other speech-related tasks.

Summary updated

BibTeX
@inproceedings{kim2022e,
  author = {Kwangyoun Kim and Felix Wu and Yifan Peng and Jing Pan and Prashant Sridhar and Kyu J. Han and Shinji Watanabe},
  title = {{E-Branchformer: Branchformer with Enhanced Merging for Speech Recognition}},
  booktitle = {SLT},
  year = {2022},
  doi = {10.1109/slt54892.2023.10022656},
}
Figure