TL;DR
E-Branchformer evolves the parallel Branchformer architecture by introducing an enhanced merging mechanism and interleaving feed-forward layers. By refining the fusion of local and global features with a simple depth-wise convolution, it achieves a new state-of-the-art on LibriSpeech (1.81% / 3.65% WER) without the need for external training data.
What Problem Does This Paper Address?
In Automatic Speech Recognition (ASR), the "encoder" is the heart of the system, responsible for turning raw audio features into meaningful hidden representations. To do this well, an encoder must capture both local context (fine-grained acoustic patterns) and global context (long-range dependencies across a sentence).
The current gold standard, the Conformer, handles this by stacking convolution and self-attention modules sequentially. A newer alternative, the Branchformer, processes these in parallel branches and merges them at the end of each block. While Branchformer is efficient, it hadn't quite dethroned the Conformer. The authors of this paper argue that the original Branchformer’s method of merging—simply concatenating the branches and applying a linear projection—is suboptimal because it ignores spatial/temporal relationships during the fusion step.
Methods
The authors propose E-Branchformer (Enhanced Branchformer), which introduces two primary architectural improvements:
- Enhanced Merge Module: Instead of a simple point-wise linear projection, E-Branchformer adds a depth-wise convolution after the branches are concatenated. This allows the model to refine the merged features by looking at neighboring time steps.
- They experimented with several merge strategies, including multiple kernels, Squeeze-and-Excitation (SE) blocks, and Conformer-style convolution modules.
- As seen in Figure 3, the most effective method (c) is a simple depth-wise convolution followed by a residual connection.
- Interleaved Feed-Forward Networks (FFNs): The original Branchformer omitted standard FFN blocks to save on computation. E-Branchformer reintroduces them, often in a "macaron" style (half-step FFNs sandwiching the main block), to increase model capacity and stability.
- Internal Language Model Estimation (ILME): During decoding, they use ILME to subtract the ASR model's internal linguistic biases, allowing for more effective "shallow fusion" with an external language model.
Datasets
The model was evaluated on the LibriSpeech dataset, a standard ASR benchmark containing approximately 1,000 hours of read English speech. The authors emphasize that their state-of-the-art results were achieved without using any external audio data for pre-training (like Wav2Vec 2.0), making it a "pure" architectural comparison.
Key Insights and Ideas
- Sequential Refinement of Parallel Branches: The core "aha" moment is that while parallel extraction is powerful, the merging process benefits significantly from a small amount of sequential temporal processing (the depth-wise convolution).
- Kernel Size Matters: Similar to the Conformer, the merge module performs best with a large kernel size ($k=31$). Small kernels ($k=3$) actually degraded performance, as shown in Table 3.
- Capacity vs. Depth: Stacking FFNs with Branchformer blocks proved more effective than simply stacking more Branchformer blocks alone. The FFNs provide a necessary point-wise refinement that the branching structure lacks.
- Efficiency: The added depth-wise convolution in the merge module adds a negligible number of parameters and computation (MACs) while providing a consistent boost in accuracy.
Key Results
E-Branchformer sets a new benchmark for models trained solely on LibriSpeech. As shown in Table 1, the "Large" version of the model achieves:
- 1.81% WER on
test-clean - 3.65% WER on
test-other
This outperforms both the original Conformer and the standard Branchformer baselines.
Ablation studies in Table 4 confirm that the depth-wise convolution is the most efficient way to improve the merge module, providing better results than more complex additions like Squeeze-and-Excitation blocks.
Limitations and Future Work
- External Data Gap: While E-Branchformer is the SOTA for models without external data, it still trails behind models like W2v-BERT that utilize massive amounts (60k+ hours) of unlabeled audio for self-supervised pre-training.
- Framework Scope: This study focused on the Attention-based Encoder-Decoder (AED) framework. The authors suggest that future work should explore applying E-Branchformer to Transducer models, which are popular for streaming ASR.
- Task Generalization: The authors are interested in testing this architecture on other speech-related tasks, such as speech enhancement, identification, and spoken language understanding (SLU).
Summary updated