On TasNet for Low-Latency Single-Speaker Speech Enhancement

Morten Kolbæk, Zheng-Hua Tan, Søren Holdt Jensen, Jesper JensenarXiv 2021 · arXiv

Abstract

This paper investigates the application of the Time-domain Audio Separation Network (TasNet), originally designed for multi-speaker separation, to the task of single-speaker speech enhancement (denoising). Through comparative analysis with standard uNet architectures, the authors demonstrate that TasNet achieves superior performance by learning an internal representation where speech and noise are more distinct than in the traditional Short-Time Fourier Transform (STFT) domain. However, the study also identifies a critical sensitivity to aliasing artifacts when using large frame hop sizes, highlighting a trade-off between computational efficiency and audio quality.

Introduction

Speech enhancement—the process of isolating a target voice from background noise—is a fundamental challenge for low-latency applications such as hearing aids and mobile communications. While TasNet has set new benchmarks in speech separation (extracting multiple voices), its efficacy for enhancement remained less explored. This study addresses three key questions: the performance gap between TasNet and established models like uNet, the underlying reasons for TasNet's success, and the impact of architectural parameters like hop size on performance.

Methods

The researchers compared two primary deep-learning architectures:

  1. TasNet: An end-to-end model comprising a linear Encoder, a Separator utilizing Temporal Convolutional Network (TCN) blocks, and a Decoder. It operates directly on time-domain waveforms.
  2. uNet: A standard encoder-decoder architecture featuring skip-connections, serving as a baseline for speech enhancement.
Architectural comparison between TasNet and uNet.

TasNet was evaluated in both non-causal (using Global Layer Normalization) and causal (using Cumulative Layer Normalization) configurations. Notably, the TasNet configuration employed a significantly larger receptive field (1.53s) compared to the uNet (0.32s).

Datasets

Evaluation utilized the Wall Street Journal (WSJ0) corpus for clean speech, combined with noise from CHiME3 and stationary speech-shaped noise (SSN). To ensure robust evaluation in multi-speaker scenarios, the authors introduced wsj0-2bal, a balanced version of the standard 2-speaker dataset that eliminates gender bias by providing an equal split of same-gender and opposite-gender pairs.

The "Inner-Domain" Advantage

A core finding of the study is that TasNet’s learned Encoder creates a representation where speech and noise overlap significantly less than in the STFT domain. Analysis showed that for a two-speaker mixture, signal overlap was reduced from 46% in STFT to just 20% in the TasNet domain. Similar reductions were observed for speech mixed with noise (SSN and BUS).

Comparison of signal overlap between channels in the TasNet learned domain versus the traditional STFT domain.

The Aliasing Trap

Despite its advantages, TasNet is highly sensitive to the "hop size" (the stride between audio frames). While the model excels with a 1ms hop size, performance degrades sharply at 32ms. The authors hypothesize that because the Separator processes rows of the inner domain independently, large hop sizes introduce aliasing artifacts—essentially decimation without filtering—that the model cannot rectify.

Key Results

TasNet consistently outperformed uNet across various noise conditions:

  • Real-world Noise: In complex "mix" noise (bus, street, etc.), TasNet achieved an SI-SDR of 12.3 dB, surpassing uNet’s 11.1 dB.
  • Competing Speakers: TasNet’s greatest advantage appeared when the "noise" was a competing speaker, reaching 16.0 dB SI-SDR compared to uNet's 13.4 dB.
  • Causality: While moving to a causal (low-latency) system reduces performance, causal TasNet remains competitive with uNet.

Conclusion and Future Work

TasNet proves to be a powerful tool for single-speaker enhancement, largely due to its superior learned representations. However, the "aliasing bottleneck" remains a challenge for reducing computational load via larger hop sizes. Future research should explore Separator architectures that operate across rows of the inner domain to mitigate aliasing and further bridge the gap between causal and non-causal performance.

Summary updated

Abstract

In recent years, speech processing algorithms have seen tremendous progress primarily due to the deep learning renaissance. This is especially true for speech separation where the time-domain audio separation network (TasNet) has led to significant improvements. However, for the related task of single-speaker speech enhancement, which is of obvious importance, it is yet unknown, if the TasNet architecture is equally successful. In this paper, we show that TasNet improves state-of-the-art also for speech enhancement, and that the largest gains are achieved for modulated noise sources such as speech. Furthermore, we show that TasNet learns an efficient inner-domain representation, where target and noise signal components are highly separable. This is especially true for noise in terms of interfering speech signals, which might explain why TasNet performs so well on the separation task. Additionally, we show that TasNet performs poorly for large frame hops and conjecture that aliasing might be the main cause of this performance drop. Finally, we show that TasNet consistently outperforms a state-of-the-art single-speaker speech enhancement system.
BibTeX
@misc{kolbk2021tasnet,
  author = {Morten Kolbæk and Zheng-Hua Tan and Søren Holdt Jensen and Jesper Jensen},
  title = {{On TasNet for Low-Latency Single-Speaker Speech Enhancement}},
  year = {2021},
  eprint = {2103.14882},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.14882},
}
Figure