E-Branchformer: Branchformer with Enhanced merging for speech recognition

Kwangyoun Kim, Felix Wu, Yifan Peng, Jing Pan, Prashant Sridhar, Kyu J. Han, Shinji Watanabe

TL;DR

E-Branchformer evolves the parallel Branchformer architecture by introducing an enhanced merging mechanism and interleaving feed-forward layers. By refining the fusion of local and global features with a simple depth-wise convolution, it achieves a new state-of-the-art on LibriSpeech (1.81% / 3.65% WER) without the need for external training data.

What Problem Does This Paper Address?

In Automatic Speech Recognition (ASR), the "encoder" is the heart of the system, responsible for turning raw audio features into meaningful hidden representations. To do this well, an encoder must capture both local context (fine-grained acoustic patterns) and global context (long-range dependencies across a sentence).

The current gold standard, the Conformer, handles this by stacking convolution and self-attention modules sequentially. A newer alternative, the Branchformer, processes these in parallel branches and merges them at the end of each block. While Branchformer is efficient, it hadn't quite dethroned the Conformer. The authors of this paper argue that the original Branchformer’s method of merging—simply concatenating the branches and applying a linear projection—is suboptimal because it ignores spatial/temporal relationships during the fusion step.

Methods

The authors propose E-Branchformer (Enhanced Branchformer), which introduces two primary architectural improvements:

  1. Enhanced Merge Module: Instead of a simple point-wise linear projection, E-Branchformer adds a depth-wise convolution after the branches are concatenated. This allows the model to refine the merged features by looking at neighboring time steps.
  2. They experimented with several merge strategies, including multiple kernels, Squeeze-and-Excitation (SE) blocks, and Conformer-style convolution modules.
  3. As seen in Figure 3, the most effective method (c) is a simple depth-wise convolution followed by a residual connection.
Fig. 3: A figure of (a) the proposed E-Branchformer block and different methods for the merge module: (b) is used in Branchformer, (c) is applying a depth-wise convolution, and (d) uses multiple convolutions with different kernel size, e.g., 31 and 3. (e) employs a squeeze-and- excitation block, and (f) replaces a depth-wise convolution with the convolution module in Conformer [28].
  1. Interleaved Feed-Forward Networks (FFNs): The original Branchformer omitted standard FFN blocks to save on computation. E-Branchformer reintroduces them, often in a "macaron" style (half-step FFNs sandwiching the main block), to increase model capacity and stability.
  2. Internal Language Model Estimation (ILME): During decoding, they use ILME to subtract the ASR model's internal linguistic biases, allowing for more effective "shallow fusion" with an external language model.

Datasets

The model was evaluated on the LibriSpeech dataset, a standard ASR benchmark containing approximately 1,000 hours of read English speech. The authors emphasize that their state-of-the-art results were achieved without using any external audio data for pre-training (like Wav2Vec 2.0), making it a "pure" architectural comparison.

Key Insights and Ideas

  • Sequential Refinement of Parallel Branches: The core "aha" moment is that while parallel extraction is powerful, the merging process benefits significantly from a small amount of sequential temporal processing (the depth-wise convolution).
  • Kernel Size Matters: Similar to the Conformer, the merge module performs best with a large kernel size ($k=31$). Small kernels ($k=3$) actually degraded performance, as shown in Table 3.
Table 3: Comparison of word error rates (WER) according to differ- ent kernel sizes of the depth-wise convolution in the merge module. The Base size model with FFN is employed for this ablation study, and we decode each models without an external LM.
  • Capacity vs. Depth: Stacking FFNs with Branchformer blocks proved more effective than simply stacking more Branchformer blocks alone. The FFNs provide a necessary point-wise refinement that the branching structure lacks.
  • Efficiency: The added depth-wise convolution in the merge module adds a negligible number of parameters and computation (MACs) while providing a consistent boost in accuracy.

Key Results

E-Branchformer sets a new benchmark for models trained solely on LibriSpeech. As shown in Table 1, the "Large" version of the model achieves:

  • 1.81% WER on test-clean
  • 3.65% WER on test-other

This outperforms both the original Conformer and the standard Branchformer baselines.

Table 1: Comparing word error rates (WER) with other published models. The external LM used in [27, 29, 59] is downloadable from ESPnet, and it is the same as what we used in all our experiments.

Ablation studies in Table 4 confirm that the depth-wise convolution is the most efficient way to improve the merge module, providing better results than more complex additions like Squeeze-and-Excitation blocks.

Table 4: Effect of different convolution-based methods added in the merge module. (b)-(f) refers to each method in Figure 3. The Base size model with FFN modules is used without an external LM.

Limitations and Future Work

  • External Data Gap: While E-Branchformer is the SOTA for models without external data, it still trails behind models like W2v-BERT that utilize massive amounts (60k+ hours) of unlabeled audio for self-supervised pre-training.
  • Framework Scope: This study focused on the Attention-based Encoder-Decoder (AED) framework. The authors suggest that future work should explore applying E-Branchformer to Transducer models, which are popular for streaming ASR.
  • Task Generalization: The authors are interested in testing this architecture on other speech-related tasks, such as speech enhancement, identification, and spoken language understanding (SLU).

Summary updated

Abstract

Conformer, combining convolution and self-attention sequentially to capture both local and global information, has shown remarkable performance and is currently regarded as the state-of-the-art for automatic speech recognition (ASR). Several other studies have explored integrating convolution and self-attention but they have not managed to match Conformer's performance. The recently introduced Branchformer achieves comparable performance to Conformer by using dedicated branches of convolution and self-attention and merging local and global context from each branch. In this paper, we propose E-Branchformer, which enhances Branchformer by applying an effective merging method and stacking additional point-wise modules. E-Branchformer sets new state-of-the-art word error rates (WERs) 1.81% and 3.65% on LibriSpeech test-clean and test-other sets without using any external training data.
BibTeX
@inproceedings{kim2022e,
  author = {Kwangyoun Kim and Felix Wu and Yifan Peng and Jing Pan and Prashant Sridhar and Kyu J. Han and Shinji Watanabe},
  title = {{E-Branchformer: Branchformer with Enhanced merging for speech recognition}},
  booktitle = {SLT},
  year = {2022},
  doi = {10.1109/slt54892.2023.10022656},
}
Figure