AMSS-Net: Audio Manipulation on User-Specified Sources with Textual\n Queries

Woosung Choi, Minseok Kim, Marco A. Martínez Ramírez, Jaehwa Chung, Soonyoung Jung

Abstract

This paper proposes a neural network that performs audio transformations to user-specified sources (e.g., vocals) of a given audio track according to a given description while preserving other sources not mentioned in the description. Audio Manipulation on a Specific Source (AMSS) is challenging because a sound object (i.e., a waveform sample or frequency bin) is 'transparent'; it usually carries information from multiple sources, in contrast to a pixel in an image. To address this challenging problem, we propose AMSS-Net, which extracts latent sources and selectively manipulates them while preserving irrelevant sources. We also propose an evaluation benchmark for several AMSS tasks, and we show that AMSS-Net outperforms baselines on several AMSS tasks via objective metrics and empirical verification.

Introduction

In digital media editing, image manipulation has become highly accessible to non-experts through AI-driven tools for tasks like inpainting or style transfer. In contrast, audio editing remains a complex domain, typically requiring professional expertise and specialized Digital Audio Workstations (DAWs). A primary reason for this difficulty is the "transparency" of audio. Unlike images, where a pixel generally belongs to a single object, a single time-frequency bin in an audio spectrogram often contains overlapping energy from multiple sources (e.g., vocals, drums, and bass).

This overlapping nature makes selective manipulation—applying an effect to one instrument without affecting others—extremely difficult. The authors define this task as Audio Manipulation on Specific Sources (AMSS). This paper introduces AMSS-Net, a framework designed to manipulate specific sounds within a mixed track using natural language queries while maintaining the integrity of the rest of the mix.

Methodology

AMSS-Net utilizes an end-to-end encoder-decoder architecture, specifically a U-Net variant, conditioned on textual descriptions. The model processes input audio $A$ and a text description $S$ to produce an estimated manipulated audio $\widehat{A'}$.

The AMSS-Net architecture, showing the flow from input audio and text description to the manipulated output.

The architecture incorporates several key components:

  1. Description Encoder ($E_{desc}$): This module uses pre-trained GloVe word embeddings and a bidirectional RNN to convert text queries (e.g., "apply low-pass to drums") into word features $w \in \mathbb{R}^{L \times E}$.
  2. Latent Source Channel (LSC) Extractor: Rather than performing explicit source separation into predefined categories, the model extracts "latent source channels." These represent abstract acoustic components that can be flexibly recombined to represent target instruments or sounds.
  3. Selective Manipulation via Point-wise Convolutional Modulation (SMPoCM): This component acts as the control center for manipulation. It employs internal gates (selective, manipulation, and input) to determine which latent channels to modify and the degree to which these modifications are integrated into the final output.
  4. Channel-wise Skip Attention (CSA): To address the limitations of standard skip connections in U-Nets, CSA uses an attention mechanism. This helps the model retain original, unmanipulated source information, ensuring that non-target audio components remain artifact-free.

Experimental Setup and Datasets

To evaluate AMSS-Net, the authors utilized the Musdb18 dataset, a standard for source separation containing 150 tracks split into vocals, drums, bass, and "other."

Since no dedicated dataset for text-based audio editing existed, the authors developed a framework for On-the-fly Triple Generation. This framework applies random digital signal processing (DSP) effects (using pysndfx) to specific stems of Musdb18 tracks, creating training triples consisting of the original mix, a text query, and the target manipulated mix.

The evaluation benchmark covers seven distinct AMSS tasks:

  • Panning: Adjusting left/right balance.
  • Volume Control: Increasing or decreasing gain.
  • Filtering: Applying low-pass or high-pass filters.
  • Dereverberation: Removing echo/reverb from specific sources.

Results and Discussion

AMSS-Net demonstrates strong performance across both objective metrics and qualitative assessments. In source separation tasks, it achieves a Signal-to-Distortion Ratio (SDR) comparable to specialized models like Meta-TasNet. For manipulation accuracy, measured by RMSE-MFCC (Root Mean Square Error of Mel-Frequency Cepstral Coefficients), AMSS-Net consistently outperformed baseline models.

Spectrogram comparisons between the original audio and various manipulated versions (kick drum, high hat, non-percussion).

Key insights from the results include:

  • Adjective Mapping: The model effectively maps intensity-related adjectives (e.g., "light," "heavy") to the mathematical parameters of DSP effects, such as filter cutoff frequencies.
  • Generational Robustness: Thanks to the CSA mechanism, AMSS-Net is resistant to "generational loss"—the accumulation of artifacts during iterative editing. This makes it suitable for progressive manipulation workflows.
  • Selective Dereverberation: The model successfully removes reverb from a target instrument while preserving the ambient characteristics of the rest of the mix.

Conclusion and Future Work

AMSS-Net represents a significant step toward making complex audio editing accessible through natural language. While the current work focuses on standard DSP effects and a limited set of instrument categories, it establishes a robust foundation for future research. Future directions include expanding the variety of supported instruments and incorporating more complex, non-linear effects like heavy distortion. Additionally, integrating generative models could further enhance the naturalness of manipulated audio for highly complex textual descriptions.

Summary updated

Abstract

This paper proposes a neural network that performs audio transformations to\nuser-specified sources (e.g., vocals) of a given audio track according to a\ngiven description while preserving other sources not mentioned in the\ndescription. Audio Manipulation on a Specific Source (AMSS) is challenging\nbecause a sound object (i.e., a waveform sample or frequency bin) is\n`transparent'; it usually carries information from multiple sources, in\ncontrast to a pixel in an image. To address this challenging problem, we\npropose AMSS-Net, which extracts latent sources and selectively manipulates\nthem while preserving irrelevant sources. We also propose an evaluation\nbenchmark for several AMSS tasks, and we show that AMSS-Net outperforms\nbaselines on several AMSS tasks via objective metrics and empirical\nverification.\n
BibTeX
@inproceedings{choi2021amss,
  author = {Woosung Choi and Minseok Kim and Marco A. Martínez Ramírez and Jaehwa Chung and Soonyoung Jung},
  title = {{AMSS-Net: Audio Manipulation on User-Specified Sources with Textual\textbackslash\{\}n Queries}},
  booktitle = {ACM Multimedia},
  year = {2021},
  doi = {10.1145/3474085.3475323},
}
Figure