Upsampling Artifacts in Neural Audio Synthesis

Jordi Pons, Santiago Pascual, Giulio Cengarle, Joan Serrà

Abstract

Recent breakthroughs in neural audio synthesis have enabled high-fidelity waveform generation at speeds far exceeding real-time. However, these models often rely on upsampling layers that introduce unintended artifacts. This paper investigates the origins of these artifacts, drawing parallels to the "checkerboard artifacts" well-known in computer vision. We demonstrate that common upsampling operators—transposed and subpixel convolutions—inherently produce tonal noise and spectral replicas. While training can mitigate these effects, interpolation-based alternatives offer a structurally safer path for high-quality audio generation.

Introduction: The "Checkerboard" of Audio

Modern feed-forward neural audio synthesizers, such as GANs and autoencoders, utilize upsampling layers to expand low-dimensional latent representations into high-resolution audio signals. While computationally efficient, these layers are prone to structural flaws. In computer vision, transposed convolutions are known to create visual checkerboard patterns. In the audio domain, these manifest as persistent high-frequency tonal artifacts. This study systematically analyzes how different upsampling architectures contribute to these unwanted sounds, even before the model begins training.

Mechanisms of Artifact Generation

The research identifies two primary sources of upsampling artifacts: structural tonal noise and spectral replication.

Tonal Artifacts and Overlap

Transposed convolutions generate tonal artifacts due to the periodic repetition of weights across time. When the filter's stride and length lead to partial overlaps, the resulting signal contains periodic energy spikes that manifest as audible tones.

Mechanism of tonal artifacts in transposed convolutions due to weight overlap and summation.

Subpixel convolutions, which use a "periodic shuffle" operation, are also susceptible. Although they avoid the overlap issues of transposed convolutions, the interleaving of weights from different filters creates a similar periodic structure that introduces high-frequency noise.

Spectral Replicas

Upsampling is essentially a bandwidth extension. Any signal present in the lower frequencies—including DC offsets introduced by biases or ReLU activations—is replicated in the higher frequencies during the upsampling process. Deep networks can compound this effect, replicating the tonal artifacts of earlier layers and turning subtle offsets into grating high-frequency tones.

Comparative Analysis of Upsampling Layers

The authors compared three main upsampling strategies:

  1. Transposed Convolutions: The current industry standard, found in models like MelGAN and Demucs. They are efficient but prone to tonal artifacts.
  2. Subpixel Convolutions: Highly efficient but also introduce tonal noise through weight interleaving.
  3. Interpolation Upsamplers: Using nearest-neighbor or linear interpolation followed by a standard convolution. These avoid tonal artifacts entirely but introduce "filtering artifacts," acting as low-pass filters that attenuate high-end frequencies.

Results and Perceptual Trade-offs

To evaluate these methods, the authors modified the Demucs music source separation model and trained it on the MUSDB18 dataset. While objective metrics like Signal-to-Distortion Ratio (SDR) favor subpixel and transposed convolutions, visual and auditory analysis reveals a different story.

Spectrogram comparison of different upsampling methods: (a) MelGAN, (b) Original Demucs, (c) Demucs with nearest neighbor upsampling, (d) Demucs with subpixel CNN. Note the horizontal lines representing tonal artifacts in (a), (b), and (d).

As shown in the spectrograms, models using transposed or subpixel convolutions (a, b, d) exhibit clear horizontal lines—tonal artifacts—across the frequency spectrum. In contrast, the nearest-neighbor approach (c) eliminates these tones, though it scores slightly lower on objective benchmarks (5.17 dB SDR vs. 5.38 dB for subpixel CNN).

Conclusion

While neural networks are adept at learning to suppress their own architectural flaws during training, the underlying structural issues of transposed and subpixel convolutions remain a liability. Tonal artifacts can still emerge in silent passages or when processing unfamiliar audio. Interpolation-based upsampling, despite a slight performance penalty in objective metrics, provides a perceptually safer alternative by inherently avoiding the generation of tonal noise.

Summary updated

Abstract

A number of recent advances in neural audio synthesis rely on up-sampling layers, which can introduce undesired artifacts. In computer vision, upsampling artifacts have been studied and are known as checkerboard artifacts (due to their characteristic visual pattern). However, their effect has been overlooked so far in audio processing. Here, we address this gap by studying this problem from the audio signal processing perspective. We first show that the main sources of upsampling artifacts are: (i) the tonal and filtering artifacts introduced by problematic upsampling operators, and (ii) the spectral replicas that emerge while upsampling. We then compare different upsampling layers, showing that nearest neighbor upsamplers can be an alternative to the problematic (but state-of-the-art) transposed and subpixel convolutions which are prone to introduce tonal artifacts.
BibTeX
@inproceedings{pons2021upsampling,
  author = {Jordi Pons and Santiago Pascual and Giulio Cengarle and Joan Serrà},
  title = {{Upsampling Artifacts in Neural Audio Synthesis}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=lRhCjUDpKs},
}
Figure