Music Enhancement via Image Translation and Vocoding

Nikhil Kandpal, Oriol Nieto, Zeyu Jin

Abstract

Consumer-grade music recordings such as those captured by mobile devices typically contain distortions in the form of background noise, reverb, and microphone-induced EQ. This paper presents a deep learning approach to enhance low-quality music recordings by combining (i) an image-to-image translation model for manipulating audio in its mel-spectrogram representation and (ii) a music vocoding model for mapping synthetically generated mel-spectrograms to perceptually realistic waveforms. We find that this approach to music enhancement outperforms baselines which use classical methods for mel-spectrogram inversion and an end-to-end approach directly mapping noisy waveforms to clean waveforms. Additionally, in evaluating the proposed method with a listening test, we analyze the reliability of common audio enhancement evaluation metrics when used in the music domain.

Introduction

With the proliferation of user-generated content, a significant portion of the world's music is now recorded on consumer-grade mobile devices in acoustically untreated environments. Unlike professional studio recordings, these amateur captures suffer from various degradations, including ambient background noise, muddy resonance, and poor frequency response due to hardware limitations.

While speech enhancement is a mature field, music enhancement presents unique challenges. First, music is often polyphonic, containing overlapping instruments and voices that complicate traditional source-based cleaning. Second, perceptual requirements differ; while reverb is generally undesirable in speech, it is often a stylistic necessity in music, making the definition of "enhancement" subjective. Finally, the specific characteristics of the recording environment and microphone are usually unknown, requiring models that can generalize across diverse distortion profiles.

Proposed Framework

This research proposes a decoupled two-stage architecture that separates the structural cleaning of audio from the synthesis of the final waveform.

Stage 1: Mel2Mel (Enhancement)

The system first converts raw audio into a mel-spectrogram, a 2D representation of frequency over time. By treating the spectrogram as an image, the framework leverages a Conditional Generative Adversarial Network (Pix2Pix) to translate "low-quality" spectrograms into "high-quality" versions. This approach rests on the hypothesis that harmonic structures and polyphonic signals are more effectively manipulated in a 2D spatial domain where visual patterns of music are distinct.

Stage 2: Musecoding (Vocoding)

Once the spectrogram is enhanced, it must be converted back into a listenable waveform. Traditional algorithms like Griffin-Lim often introduce metallic or robotic artifacts. To ensure perceptual realism, the authors employ DiffWave, a Denoising Diffusion Probabilistic Model (DDPM). Trained specifically on high-quality music, this vocoder acts as a universal synthesizer, mapping the synthetic mel-spectrograms to high-fidelity audio while remaining robust to any residual artifacts from the GAN stage.

Experimental Setup

Models were trained using the Medley-solos-DB dataset, which consists of over 21,000 professional recordings of individual instruments. To simulate real-world amateur recordings, the authors developed a generative simulation pipeline to "degrade" the clean data:

  • Acoustic Simulation: Convolving audio with Room Impulse Responses (RIR) to mimic varied room acoustics.
  • Noise Injection: Adding background noise at varying Signal-to-Noise Ratios (SNR).
  • Hardware Emulation: Applying random 4-band equalization (EQ) to simulate the frequency response of inexpensive microphones.

Results and Discussion

The proposed method was evaluated against several baselines, including Demucs (a popular end-to-end waveform model) and a version of the pipeline using classical Griffin-Lim inversion. A large-scale listening test involving 211 participants on Amazon Mechanical Turk was used to determine the Mean Opinion Score (MOS).

Model MOS (Higher is Better)
Clean (Ground Truth) 4.39
Mel2Mel + Diffwave (Proposed) 4.06
Demucs (End-to-End Baseline) 3.39
Mel2Mel + Griffin-Lim 3.01
No Enhancement 2.85

The proposed framework achieved a MOS of 4.06, significantly outperforming the end-to-end Demucs baseline and approaching the quality of professional ground-truth recordings.

The Metric Gap

A critical finding of this study is the unreliability of standard objective metrics (such as fwSSNR or L1 distance) in the music domain. The authors observed that these metrics often favored models that produced "robotic" or unnatural sounds if they were mathematically closer to the source, whereas human listeners overwhelmingly preferred the perceptually richer output of the diffusion-based vocoder. This highlights a significant need for new, music-specific perceptual metrics.

Conclusion and Future Work

The decoupling of enhancement and synthesis proves to be a powerful strategy for music restoration. However, challenges remain in dereverberation; the model occasionally struggled to distinguish between desirable stylistic reverb and unwanted room resonance. Future research will focus on improving the model's performance on pre-cleaned samples to avoid introducing unnecessary artifacts and developing more robust perceptual metrics to better align automated evaluation with human hearing.

Summary updated

Abstract

Consumer-grade music recordings such as those captured by mobile devices typically contain distortions in the form of background noise, reverb, and microphone-induced EQ. This paper presents a deep learning approach to enhance low-quality music recordings by combining (i) an image-to-image translation model for manipulating audio in its mel-spectrogram representation and (ii) a music vocoding model for mapping synthetically generated mel-spectrograms to perceptually realistic waveforms. We find that this approach to music enhancement outperforms baselines which use classical methods for mel-spectrogram inversion and an end-to-end approach directly mapping noisy waveforms to clean waveforms. Additionally, in evaluating the proposed method with a listening test, we analyze the reliability of common audio enhancement evaluation metrics when used in the music domain.
BibTeX
@inproceedings{kandpal2022music,
  author = {Nikhil Kandpal and Oriol Nieto and Zeyu Jin},
  title = {{Music Enhancement via Image Translation and Vocoding}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=V6PrHIiju9},
}
Figure