Multi-Scale Group Transformer for Long Sequence Modeling in Speech Separation

Yucheng Zhao, Chong Luo, Zheng-Jun Zha, Wenjun Zeng

Introduction

In the field of speech separation—often referred to as the "cocktail party problem"—the goal is to isolate individual speakers from a mixed audio signal. While time-domain methods have recently outperformed traditional frequency-based approaches, they require processing extremely long sequences (e.g., over 4,000 frames for a 4-second clip). Standard Transformer architectures, despite their strength in modeling long-range dependencies, suffer from quadratic computational complexity ($O(n^2)$), making them impractical for such high-resolution audio tasks. Researchers from Microsoft and USTC have addressed this bottleneck by introducing the Multi-Scale Group Transformer (MSGT), a model that achieves linear or log-linear complexity while maintaining high separation quality.

The Computational Challenge

Time-domain speech separation performs best with very short audio frames (approx. 2ms). However, this granularity results in massive sequence lengths that overwhelm the memory and processing capabilities of standard self-attention mechanisms. Previous solutions relied on Recurrent Neural Networks (RNNs) or Temporal Convolutional Networks (TCNs), but these often struggle to capture the global context necessary to distinguish overlapping voices over extended periods. The MSGT aims to bring the modeling power of Transformers to this domain without the prohibitive cost.

The Multi-Scale Group Transformer (MSGT) Architecture

The MSGT architecture relies on two core innovations to manage long sequences:

  1. Group Self-Attention (GSA): Instead of global attention, the sequence is partitioned into smaller, non-overlapping groups. Self-attention is restricted to within these groups, reducing complexity to $O(g \cdot n)$, where $g$ is a fixed group size.
  2. Multi-Scale Fusion: To compensate for the limited receptive field of GSA, the model employs a hierarchical structure. Audio features are down-sampled into multiple scales. At high resolutions, GSA captures fine-grained local details; at lower resolutions, the same fixed group size covers a larger temporal span, effectively capturing global dependencies.
Overview of the Multi-Scale Group Transformer (MSGT) architecture, utilizing hierarchical feature maps and group transformers.

Fusion Strategies: Dense vs. Light

The authors integrated the MSGT module into the Conv-TasNet framework, replacing the original TCN-based separator. They explored two primary methods for communicating information across these hierarchical scales:

  • Light Fusion: This version fuses features only between adjacent scales. It maintains strictly linear complexity and offers a significant speedup (3.9x) over traditional TCNs.
  • Dense Fusion: This version enables communication across all scales simultaneously. While it has log-linear complexity, it provides superior separation performance by ensuring more robust global context integration.
Comparison of (a) Dense Fusion and (b) Light Fusion architectures for cross-scale feature communication.

Experimental Results

The model was evaluated on the WSJ0-2mix (clean) and WHAM! (noisy) benchmarks using the Scale-Invariant Signal-to-Distortion Ratio improvement (SI-SDRi) metric.

  • Performance Gains: On the clean WSJ0-2mix dataset, MSGT-TasNet (Dense) achieved 17.0 dB, a marked improvement over the 15.3 dB of the original Conv-TasNet.
  • Surpassing Theoretical Benchmarks: On the noisy WHAM! dataset, the Dense version reached 13.1 dB, becoming the first method to surpass the performance of the Ideal Ratio Mask (IRM), a theoretical benchmark previously considered a performance ceiling.
  • Efficiency: The MSGT maintains nearly constant memory usage and high throughput even as sequence lengths exceed 4,000 frames, effectively solving the "computational wall" faced by standard Transformers.

Conclusion and Future Work

The Multi-Scale Group Transformer demonstrates that with clever hierarchical grouping and fusion, Transformers can be both efficient and highly effective for long-sequence audio tasks. While the "Dense" fusion offers the best performance, the "Light" version provides an excellent trade-off for real-time applications. Future research may involve using Neural Architecture Search (NAS) to further optimize these multi-scale structures and integrating MSGT with other advanced separation techniques to push the state-of-the-art even further.

Summary updated

Abstract

In this paper, we introduce Transformer to the time-domain methods for single-channel speech separation. Transformer has the potential to boost speech separation performance because of its strong sequence modeling capability. However, its computational complexity, which grows quadratically with the sequence length, has made it largely inapplicable to speech applications. To tackle this issue, we propose a novel variation of Transformer, named multi-scale group Transformer (MSGT). The key ideas are group self-attention, which significantly reduces the complexity, and multi-scale fusion, which retains Transform's ability to capture long-term dependency. We implement two versions of MSGT with different complexities, and apply them to a well-known time-domain speech separation method called Conv-TasNet. By simply replacing the original temporal convolutional network (TCN) with MSGT, our approach called MSGT-TasNet achieves a large gain over Conv-TasNet on both WSJ0-2mix and WHAM! benchmarks. Without bells and whistles, the performance of MSGT-TasNet is already on par with the SOTA methods.
BibTeX
@inproceedings{zhao2020multi,
  author = {Yucheng Zhao and Chong Luo and Zheng-Jun Zha and Wenjun Zeng},
  title = {{Multi-Scale Group Transformer for Long Sequence Modeling in Speech Separation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/450},
}
Figure