DeFTAN-II: Efficient Multichannel Speech Enhancement with Subgroup Processing

Dongheon Lee, Jung‐Woo ChoiIEEE/ACM Trans. ASLP 2023

TL;DR

DeFTAN-II is a high-efficiency transformer model for multichannel speech enhancement that introduces "subgroup processing" to capture local acoustic details while slashing computational costs. It outperforms current state-of-the-art models like TF-GridNet on standard benchmarks while requiring up to 7x fewer computations and significantly less memory.

What Problem Does This Paper Address?

Multichannel speech enhancement aims to clean up a "reference" microphone signal by leveraging spatial information from an array of microphones. While Transformer architectures have recently dominated this field due to their ability to model long-range (global) relationships, they suffer from two major drawbacks:

  1. High Complexity: Self-attention mechanisms scale quadratically with sequence length, leading to massive memory and compute requirements.
  2. Local Blindness: Transformers often struggle to capture fine-grained local relations (like specific frequency harmonics or short-term speech transients) that are easily handled by Convolutional Neural Networks (CNNs).

Previous attempts to fix this, like DenseNets, often lead to a "linear explosion" of feature map sizes, making them impractical for resource-constrained hardware.

Methods

The authors propose DeFTAN-II, which integrates CNN-like local processing into a Transformer framework using a strategy called Subgroup Processing. The architecture is built on three novel components:

  • Split Dense Block (SDB): Instead of processing all features at once, the SDB splits them into $G$ subgroups. It processes the first group, concatenates the result with the second group, and repeats. This "sequential reuse" allows the model to extract deep spatial features without the memory bloat of standard dense blocks.
  • Convolutional Efficient Attention (CEA): This is an asymmetric attention mechanism. It creates "locally emphasized" features (using CNNs) for the Keys and Queries, while keeping the Values original. This "cross-attention" between local and global features is more efficient and effective than standard self-attention.
  • Dual-Path Feedforward Network (DPFN): In the transformer's feedforward layer, the model splits the data into two paths. One path uses dilated convolutions to expand the receptive field locally, while the other remains a simple projection. This balances local context with global feature aggregation.

Datasets

The model was trained and evaluated on three rigorous simulated datasets and one real-world setup:

  • Spatialized WSJCAM0: Focuses on high reverberation environments.
  • Spatialized DNS Challenge 2020: Focuses on non-stationary noise and low Signal-to-Noise Ratios (SNR).
  • L3DAS22 Challenge: A benchmark for 3D office environments using Ambisonic microphone arrays.
  • Real-world Office Recording: To test generalization, the authors recorded speech in a physical office using a ReSpeaker Mic Array, evaluating the model without any fine-tuning on this specific hardware.

Key Insights and Ideas

  • Sequential is More Efficient than Parallel: The Split Dense Block proves that you don't need to stack every feature map to get "dense" benefits. Sequential concatenation and shrinking maintain performance while keeping the channel dimension under control.
  • The Power of Asymmetry: By using different processing for Keys/Queries vs. Values in the attention layer, the model effectively "looks" for local patterns to decide how to weight the global information.
  • Generalization via Architecture, not Data: One of the most impressive findings was the model's ability to adapt to a real-world microphone array with a different geometry (2.7 cm radius) than the one it was trained on (10 cm radius) without needing to be retrained.

Key Results

DeFTAN-II sets a new bar for the "performance-per-watt" category in speech enhancement:

  • Superior Quality: On the WSJCAM0 dataset, DeFTAN-II (large) achieved an SI-SDR of 20.6 dB, significantly beating the previous SOTA, TF-GridNet (17.5 dB).
  • Massive Speedup: The "base" version of DeFTAN-II achieves better results than TF-GridNet while using only 64.5 G MAC/s compared to TF-GridNet’s 462.2 G MAC/s—a roughly 7-fold reduction in computational load.
  • Real-World Robustness: In real-office tests, the model reduced the Word Error Rate (WER) from 53.3% (unprocessed) to 5.2%, proving it isn't just "overfitting" to simulations.

Limitations and Future Work

  • Non-Causality: The current architecture is non-causal, meaning it needs to "see" the entire audio clip (or a large future buffer) to work. This makes it unsuitable for ultra-low-latency applications like live hearing aids without further modification.
  • Array Geometry: While the model showed flexibility with different circular arrays, the authors suggest more research is needed to see how well it adapts to completely different geometries (e.g., linear vs. spherical arrays).
  • Complexity vs. Real-time: Even with the 7x reduction, the "large" model (124 G MAC/s) might still be heavy for some mobile processors, suggesting a need for even smaller "tiny" versions.

Summary updated

Abstract

In this work, we present DeFTAN-II, an efficient multichannel speech enhancement model based on transformer architecture and subgroup processing. Despite the success of transformers in speech enhancement, they face challenges in capturing local relations, reducing the high computational complexity, and lowering memory usage. To address these limitations, we introduce subgroup processing in our model, combining subgroups of locally emphasized features with other subgroups containing original features. The subgroup processing is implemented in several blocks of the proposed network. In the proposed split dense blocks extracting spatial features, a pair of subgroups is sequentially concatenated and processed by convolution layers to effectively reduce the computational complexity and memory usage. For the F- and T-transformers extracting temporal and spectral relations, we introduce cross-attention between subgroups to identify relationships between locally emphasized and non-emphasized features. The dual-path feedforward network then aggregates attended features in terms of the gating of local features processed by dilated convolutions. Through extensive comparisons with state-of-the-art multichannel speech enhancement models, we demonstrate that DeFTAN-II with subgroup processing outperforms existing methods at significantly lower computational complexity. Moreover, we evaluate the model's generalization capability on real-world data without fine-tuning, which further demonstrates its effectiveness in practical scenarios.
BibTeX
@article{lee2023deftan,
  author = {Dongheon Lee and Jung‐Woo Choi},
  title = {{DeFTAN-II: Efficient Multichannel Speech Enhancement with Subgroup Processing}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2023},
}
Figure