End-to-end Music Remastering System Using Self-supervised and Adversarial Training

Junghyun Koo, Seungryeol Paik, Kyogu Lee

Abstract

This paper presents an end-to-end neural network system designed for audio remastering, capable of transferring the professional mastering style of a reference track onto an input song. By integrating self-supervised contrastive learning with adversarial training, the system learns to adjust loudness, stereo width, and tonal balance without requiring original unmastered recordings for training. Experimental results demonstrate that the proposed model effectively matches target mastering styles while preserving the integrity of the underlying musical content.

Introduction

Audio mastering is the final stage of music production, where engineers adjust tone, spatial dynamics, and loudness to ensure a track sounds professional across all playback systems. Remastering specifically involves updating older recordings to meet modern sonic standards. Traditionally, this is a high-barrier process requiring specialized equipment and years of experience.

Existing "intelligent" systems often attempt to estimate settings for specific software plug-ins. However, these methods frequently fail when the original "dry" (unprocessed) studio tracks are unavailable—a common issue with vintage recordings. This paper proposes a solution for end-to-end remastering that operates directly from audio to audio by learning the "style" of a reference track.

Methodology

The system architecture consists of four primary components working in concert to perform style transfer:

  1. Mastering Effects Manipulator: To overcome the scarcity of unmastered data, this module applies random chains of EQ, stereo imaging, and compression to existing songs, creating synthetic training pairs.
  2. Music Effects Encoder: Pre-trained using contrastive learning (SimCLR), this module extracts a high-level representation of a song's "sonic fingerprint," including timbre, mood, and spatial characteristics.
  3. Mastering Cloner: Based on the Wave-U-Net architecture, this engine transforms the input audio. It utilizes Feature-wise Linear Modulation (FiLM) to inject style features from the encoder into the network.
  4. Projection Discriminator: Employed in a Generative Adversarial Network (GAN) setup, this component ensures the output matches the style embedding provided by the encoder, maintaining high-fidelity results.
The proposed end-to-end remastering architecture, featuring self-supervised style encoding and adversarial training.

To maintain high fidelity, the authors implemented specific anti-aliasing techniques within the U-Net and introduced a specialized RMS loss function to prioritize accurate loudness matching.

Experimental Results

The researchers evaluated their full model (Model 3) against several baseline iterations. Quantitative analysis focused on loudness matching ($\Delta$RMS), stereo width ($\Delta$RMS-side), and signal-to-noise ratios.

Quantitative comparison of different model iterations across loudness, stereo width, and intelligibility metrics.

As shown in the results, Model 3 achieved the lowest $\Delta$RMS (0.0212) and $\Delta$RMS-side (0.0396), indicating superior performance in matching target loudness and spatial characteristics. Qualitative visual analysis confirmed the model's ability to shift a track's LUFS (Loudness Units Full Scale) to match a target reference precisely. Furthermore, in MUSHRA listening tests, 17 experts gave the model a median score of 0.71, significantly outperforming basic anchors and proving the system can alter mastering style without distorting musical content.

Conclusion and Future Work

The proposed system successfully automates the complex task of audio remastering through a self-supervised, end-to-end approach. While highly effective at volume and spatial manipulation, the authors noted that spectral accuracy above 1kHz can still be improved. Future work will focus on increasing robustness to noise for restoring vintage recordings and introducing user-controllable modules to allow for manual intensity adjustments of the mastering style.

Summary updated

Abstract

Mastering is an essential step in music production, but it is also a challenging task that has to go through the hands of experienced audio engineers, where they adjust tone, space, and volume of a song. Remastering follows the same technical process, in which the context lies in mastering a song for the times. As these tasks have high entry barriers, we aim to lower the barriers by proposing an end-to-end music remastering system that transforms the mastering style of input audio to that of the target. The system is trained in a self-supervised manner, in which released pop songs were used for training. We also anticipated the model to generate realistic audio reflecting the reference's mastering style by applying a pre-trained encoder and a projection discriminator. We validate our results with quantitative metrics and a subjective listening test and show that the model generated samples of mastering style similar to the target.
BibTeX
@inproceedings{koo2022end,
  author = {Junghyun Koo and Seungryeol Paik and Kyogu Lee},
  title = {{End-to-end Music Remastering System Using Self-supervised and Adversarial Training}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9746389},
}
Figure