DeepSpace: Dynamic Spatial and Source Cue Based Source Separation for Dialog Enhancement

Aaron Master, Lie Lu, Jonas Samuelsson, Heidi-Maria Lehtonen, Scott Norcross, Nathan Swedlow, Audrey Howard

Introduction

In modern home entertainment, a common frustration for viewers is the inability to clearly hear dialog over loud background music or sound effects. This challenge has led to the development of Dialog Enhancement (DE) technologies. While "guided" DE—where dialog and background tracks are provided separately—is straightforward, most consumer content is delivered as a single downmixed stereo or multichannel stream. To improve clarity in these scenarios, systems must perform "unguided" source separation, un-mixing the audio in real-time.

DeepSpace, a system developed by Dolby, addresses this by combining traditional spatial filtering with deep learning-based source separation. Unlike standard Speech Enhancement (SE) models designed for telecommunications, DeepSpace is optimized for high-fidelity entertainment content, allowing for significant dialog boosts while maintaining the cinematic integrity of the original mix.

The Challenge: DE vs. SE

The researchers distinguish Dialog Enhancement from traditional Speech Enhancement (SE) used in applications like VoIP. Standard SE systems typically operate on mono, 16kHz audio and aim to eliminate background noise entirely. In contrast, entertainment content is usually stereo or 5.1 surround sound sampled at 48kHz. In this context, the background audio is not "noise" to be removed, but an essential part of the artistic experience. The goal of DE is to create a dialog-boosted signal where the original mix is preserved but the balance is shifted, requiring a more nuanced approach to separation.

System Architecture

DeepSpace utilizes a hybrid architecture that exploits two distinct types of information: spatial cues and source cues. The pipeline consists of three primary components:

  1. Spatio-Level Filtering (SLF): This stage exploits spatial cues by identifying sounds based on their position in the stereo field. For example, dialog is often center-panned, while music and effects may be wide-panned. SLF uses these panning and level characteristics to provide an initial layer of separation.
  2. U-NetFB: This deep learning model focuses on source cues—the specific spectral characteristics of human speech. Rather than using raw frequency bins, U-NetFB utilizes frequency band energies to generate a "softmask" that isolates speech components. This allows the system to identify voices even when they are panned similarly to background elements.
  3. Classification & Gating: To prevent the system from accidentally boosting background noises that mimic speech (such as certain musical instruments), a dedicated neural network monitors the audio. If dialog is not detected, the system "gates" or shuts off the enhancement, ensuring the background remains clean.

By passing audio through the spatial filter first and then the deep learning denoiser, DeepSpace achieves a synergy that reduces the "musical noise" artifacts often associated with aggressive deep learning models.

Experimental Methodology

The system was evaluated using two primary datasets:

  • MUSHRA Test Set: A collection of real-world broadcast signals from movies and TV shows, covering various genres, speaker genders, and dialog-to-noise ratios.
  • Large-Scale Stereo Dataset: 384 stereo clips (10 seconds each) with ground-truth clean dialog. This dataset included complex panning scenarios, including center-panned (60%), off-center (30%), and moving speakers (10%).

Key Insights and Results

The Metric Gap

One of the paper's most significant findings is the failure of standard automated metrics. The researchers discovered that metrics like PESQ (Perceptual Evaluation of Speech Quality) and STOI (Short-Time Objective Intelligibility) have zero meaningful correlation with human ratings for dialog enhancement. This is because these metrics penalize the presence of background audio, which is a desired component in a cinematic mix. This highlights a critical need for new, DE-specific objective metrics.

Subjective Performance

In MUSHRA tests involving expert listening panels, DeepSpace significantly outperformed state-of-the-art SE models such as DPCRN and NSNet2. In Absolute Category Rating (ACR) tests, the researchers found that a 12dB dialog boost using DeepSpace achieved overall quality scores nearly identical to the original professional mix. While the boost introduced minor speech distortion, the improved intelligibility and "background non-intrusiveness" resulted in a superior listener experience.

Robustness

DeepSpace demonstrated high robustness across challenging scenarios, including off-center panned dialog and highly reverberant speech—conditions that often cause simpler spatial filters to fail.

Conclusion and Future Work

DeepSpace represents a significant step forward in unguided dialog enhancement by successfully merging spatial and source-based separation. While the current iteration focuses on stereo audio, the authors note that extending the system to 5.1 surround sound is a logical progression. Additionally, the development of new objective metrics that better align with human perception remains a high priority for the field to facilitate faster iteration on DE technologies.

Summary updated

Abstract

Dialog Enhancement (DE) is a feature which allows a user to increase the level of dialog in TV or movie content relative to nondialog sounds. When only the original mix is available, DE is "unguided," and requires source separation. In this paper, we describe the DeepSpace system, which performs source separation using both dynamic spatial cues and source cues to support unguided DE. Its technologies include spatio-level filtering (SLF) and deep-learning based dialog classification and denoising. Using subjective listening tests, we show that DeepSpace demonstrates significantly improved overall performance relative to state-of-the-art systems available for testing. We explore the feasibility of using existing automated metrics to evaluate unguided DE systems.
BibTeX
@inproceedings{master2023deepspace,
  author = {Aaron Master and Lie Lu and Jonas Samuelsson and Heidi-Maria Lehtonen and Scott Norcross and Nathan Swedlow and Audrey Howard},
  title = {{DeepSpace: Dynamic Spatial and Source Cue Based Source Separation for Dialog Enhancement}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=lySMStJtyD},
}
Figure