Abstract
Recent breakthroughs in neural audio synthesis have enabled high-fidelity waveform generation at speeds far exceeding real-time. However, these models often rely on upsampling layers that introduce unintended artifacts. This paper investigates the origins of these artifacts, drawing parallels to the "checkerboard artifacts" well-known in computer vision. We demonstrate that common upsampling operators—transposed and subpixel convolutions—inherently produce tonal noise and spectral replicas. While training can mitigate these effects, interpolation-based alternatives offer a structurally safer path for high-quality audio generation.
Introduction: The "Checkerboard" of Audio
Modern feed-forward neural audio synthesizers, such as GANs and autoencoders, utilize upsampling layers to expand low-dimensional latent representations into high-resolution audio signals. While computationally efficient, these layers are prone to structural flaws. In computer vision, transposed convolutions are known to create visual checkerboard patterns. In the audio domain, these manifest as persistent high-frequency tonal artifacts. This study systematically analyzes how different upsampling architectures contribute to these unwanted sounds, even before the model begins training.
Mechanisms of Artifact Generation
The research identifies two primary sources of upsampling artifacts: structural tonal noise and spectral replication.
Tonal Artifacts and Overlap
Transposed convolutions generate tonal artifacts due to the periodic repetition of weights across time. When the filter's stride and length lead to partial overlaps, the resulting signal contains periodic energy spikes that manifest as audible tones.
Subpixel convolutions, which use a "periodic shuffle" operation, are also susceptible. Although they avoid the overlap issues of transposed convolutions, the interleaving of weights from different filters creates a similar periodic structure that introduces high-frequency noise.
Spectral Replicas
Upsampling is essentially a bandwidth extension. Any signal present in the lower frequencies—including DC offsets introduced by biases or ReLU activations—is replicated in the higher frequencies during the upsampling process. Deep networks can compound this effect, replicating the tonal artifacts of earlier layers and turning subtle offsets into grating high-frequency tones.
Comparative Analysis of Upsampling Layers
The authors compared three main upsampling strategies:
- Transposed Convolutions: The current industry standard, found in models like MelGAN and Demucs. They are efficient but prone to tonal artifacts.
- Subpixel Convolutions: Highly efficient but also introduce tonal noise through weight interleaving.
- Interpolation Upsamplers: Using nearest-neighbor or linear interpolation followed by a standard convolution. These avoid tonal artifacts entirely but introduce "filtering artifacts," acting as low-pass filters that attenuate high-end frequencies.
Results and Perceptual Trade-offs
To evaluate these methods, the authors modified the Demucs music source separation model and trained it on the MUSDB18 dataset. While objective metrics like Signal-to-Distortion Ratio (SDR) favor subpixel and transposed convolutions, visual and auditory analysis reveals a different story.
As shown in the spectrograms, models using transposed or subpixel convolutions (a, b, d) exhibit clear horizontal lines—tonal artifacts—across the frequency spectrum. In contrast, the nearest-neighbor approach (c) eliminates these tones, though it scores slightly lower on objective benchmarks (5.17 dB SDR vs. 5.38 dB for subpixel CNN).
Conclusion
While neural networks are adept at learning to suppress their own architectural flaws during training, the underlying structural issues of transposed and subpixel convolutions remain a liability. Tonal artifacts can still emerge in silent passages or when processing unfamiliar audio. Interpolation-based upsampling, despite a slight performance penalty in objective metrics, provides a perceptually safer alternative by inherently avoiding the generation of tonal noise.
Summary updated