Music Enhancement via Image Translation and Vocoding

Nikhil Kandpal, Oriol Nieto, Zeyu Jin

TL;DR

This paper introduces a two-stage framework for upgrading low-quality amateur music recordings to professional-grade audio. By treating audio mel-spectrograms as images and applying image-to-image translation (Mel2Mel) followed by a diffusion-based vocoder (Musecoding), the authors achieve results that significantly outperform traditional end-to-end waveform mapping and classical signal processing techniques.

What Problem Does This Paper Address?

With the explosion of user-generated content, much of the world's music is now recorded on consumer-grade mobile devices in acoustically "untreated" environments. These recordings suffer from background noise, muddy resonance, and poor frequency response (EQ).

While speech enhancement is a well-studied field, music presents unique challenges:

  1. Polyphony: Unlike speech, music often contains multiple overlapping instruments and voices, making source-based cleaning difficult.
  2. Perceptual Differences: In speech, reverb is almost always a nuisance. In music, a certain amount of reverb is often stylistically desirable, making "enhancement" a subjective target.
  3. Unknown Variables: The specific microphone characteristics and room acoustics of an amateur recording are usually unknown, requiring a model that can implicitly infer and correct these distortions.

Methods

The authors propose a decoupled architecture that separates the "cleaning" of the audio's structure from the "synthesis" of the final sound.

  1. Mel2Mel (Enhancement): The system first converts audio into a mel-spectrogram— a visual representation of frequency over time. They treat this spectrogram as an image and use a Conditional GAN (Pix2Pix) to translate "low-quality" images into "high-quality" ones. The authors hypothesize that it is easier to fix polyphonic signals in this 2D domain where harmonic structures are more apparent.
  2. Musecoding (Vocoding): Once the spectrogram is cleaned, it must be turned back into a listenable waveform. Instead of using classical algorithms like Griffin-Lim (which often introduce "robotic" artifacts), they use DiffWave, a Denoising Diffusion Probabilistic Model (DDPM). This model is trained specifically on high-quality music to ensure the final output sounds perceptually realistic and is robust to any synthetic artifacts left over by the GAN.

Datasets

The models were trained and evaluated using the Medley-solos-DB dataset, which contains over 21,000 professional recordings of single instruments.

To create a training set, the authors developed a generative simulation process to turn these clean recordings into "dirty" ones by:

  • Convolving audio with Room Impulse Responses (RIR) to simulate varied room acoustics.
  • Adding background noise at various Signal-to-Noise Ratios (SNR).
  • Applying random 4-band equalization to mimic cheap microphone hardware.

Key Insights and Ideas

  • Decoupling is Key: By separating enhancement (Mel2Mel) from synthesis (Musecoding), the vocoder can be trained once on clean data. This makes it a "universal" music synthesizer that doesn't need to be retrained for every new type of noise or distortion.
  • Audio as Images: The success of the Pix2Pix architecture suggests that the spatial patterns in a spectrogram (like the horizontal lines of harmonics) are effectively handled by standard computer vision techniques.
  • The "Metric Gap": A major takeaway is that standard objective metrics (like fwSSNR or L1 distance) are poor proxies for how humans actually perceive music quality. For instance, some metrics rated "robotic" sounding baseline outputs higher than the proposed method, even though human listeners overwhelmingly preferred the latter.

Key Results

The authors conducted a large-scale listening test with 211 participants on Amazon Mechanical Turk to calculate a Mean Opinion Score (MOS):

Model MOS (Higher is Better)
Clean (Ground Truth) 4.39
Mel2Mel + Diffwave (Proposed) 4.06
Mel2Mel + Griffin-Lim 3.01
Demucs (End-to-End Baseline) 3.39
No Enhancement 2.85

The proposed method achieved a score of 4.06, coming remarkably close to the original professional recordings. It significantly outperformed Demucs, a popular end-to-end model originally designed for music source separation.

Limitations and Future Work

  • Dereverberation: While the model excelled at removing noise and fixing EQ, the ablation studies showed that "dereverberation" (removing room echo) remains a difficult challenge. The models sometimes struggled to distinguish between "good" stylistic reverb and "bad" room resonance.
  • Training Mismatch: The authors noted that if a model is trained only on noisy samples, it might struggle when presented with a sample that is already clean, occasionally introducing artifacts where none were needed.
  • Metric Development: The poor correlation between objective metrics and human scores highlights a desperate need for new, music-specific perceptual metrics similar to those that exist for speech (like DNSMOS).

Summary updated

Abstract

Consumer-grade music recordings such as those captured by mobile devices typically contain distortions in the form of background noise, reverb, and microphone-induced EQ. This paper presents a deep learning approach to enhance low-quality music recordings by combining (i) an image-to-image translation model for manipulating audio in its mel-spectrogram representation and (ii) a music vocoding model for mapping synthetically generated mel-spectrograms to perceptually realistic waveforms. We find that this approach to music enhancement outperforms baselines which use classical methods for mel-spectrogram inversion and an end-to-end approach directly mapping noisy waveforms to clean waveforms. Additionally, in evaluating the proposed method with a listening test, we analyze the reliability of common audio enhancement evaluation metrics when used in the music domain.
BibTeX
@inproceedings{kandpal2022music,
  author = {Nikhil Kandpal and Oriol Nieto and Zeyu Jin},
  title = {{Music Enhancement via Image Translation and Vocoding}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=V6PrHIiju9},
}

Loading related content...

Figure