In deep learning architectures for image synthesis, super-resolution, and dense pixel prediction, a neural network must eventually restore low-resolution feature maps back to high-resolution outputs. Historically, this spatial expansion was achieved using transposed convolutions (sometimes called deconvolutions).
However, as noted by Kai Zhang et al. in the AIM 2019 Challenge on Constrained Super-Resolution: Methods and Results, the upsampling stage frequently acts as a primary computational bottleneck. More critically, transposed convolutions suffer from a structural flaw: when the convolutional kernel size is not perfectly divisible by the stride, the filters overlap unevenly. This periodic, mathematical asymmetry creates a distinct, unnatural grid pattern known as a checkerboard artifact.
To build faster networks that generate pristine, artifact-free images, researchers have shifted away from transposed convolutions toward two core mechanisms: pixel shuffling and resize-convolution upsampling.
The Mechanics of Pixel Shuffling
Instead of stretching a spatial grid and inserting zeros (as a transposed convolution does), pixel shuffling (or sub-pixel convolution) reverses the problem framing. It shifts the computational burden entirely to standard convolutions operating on the dense, low-resolution feature map, and handles the spatial expansion via a deterministic tensor rearrangement.
Imagine you want to upscale an image by a factor of 2. Instead of generating a single high-resolution feature map directly, a standard convolution is trained to output 4 distinct low-resolution feature maps in the channel dimension. Each of these 4 channels represents a specific sub-pixel offset (top-left, top-right, bottom-left, bottom-right) of the target high-resolution image. A deterministic "depth-to-space" operation then shuffles these channels into a single, higher-resolution grid.
Low-Res Feature Map Channel-to-Spatial High-Res Output
(H x W x C*4) Rearrangement (2H x 2W x C)
[ C1 C2 ] -------------> [ C1 C2 ]
[ C3 C4 ] [ C3 C4 ]
As Xiaobiao Du et al. demonstrate in Two-Stage Network for Single Image Super-Resolution, this approach learns an upscaling mapping that effectively rearranges channel-wise information into spatial dimensions. Because the convolutions occur entirely on a dense, low-resolution grid, the operation is highly efficient.
Andrey Ignatov et al. leveraged this efficiency in the Real-Time Video Super-Resolution on Smartphones With Deep Learning, Mobile AI 2021 Challenge: Report, showing that depth-to-space upsampling reduces computational overhead on mobile hardware while avoiding common upsampling artifacts. Similarly, Haojie Liu et al. in Deep Image Compression via End-to-End Learning confirmed that pixel-shuffle layers prevent checkerboard artifacts during the decoding phase of image reconstruction.
The Resize-Convolution Alternative
Another intuitive way to eliminate checkerboard artifacts is to separate the geometric upscaling from the feature learning entirely. This technique, known as resize-convolution, first applies a deterministic, smooth interpolation (such as bilinear or nearest-neighbor upsampling) to scale the feature map, followed by a standard stride-1 convolution.
Because the initial upsampling is uniform across the entire grid, there is no uneven kernel overlap. The subsequent convolution simply refines the smoothly stretched features.
Bernhard Schölkopf et al. utilized this mechanism in EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis, using nearest-neighbor upsampling followed by a standard convolutional layer to avoid checkerboard patterns.
In the context of image synthesis, Patsorn Sangkloy et al. demonstrated in Scribbler: Controlling Deep Image Synthesis with Sketch and Color that substituting transposed convolutions with bilinear upsampling mitigates artifacts and stabilizes the training of deep generative networks.
This architectural refinement is so robust at removing artificial patterns that in The Point Where Reality Meets Fantasy: Mixed Adversarial Generators for Image Splice Detection, Vladimir V. Kniaz used it to ensure that image manipulation detectors learn genuine forgery signatures rather than artifacts introduced by the generator's own upsampling layers.
Hybrid Strategies: Balancing Sharpness and Smoothness
While pixel shuffling provides sharp high-frequency details, it can occasionally introduce high-frequency noise if the channels are not perfectly coordinated. Conversely, resize-convolution provides smooth transitions but can blur fine edges. Modern architectures frequently combine both to achieve optimal reconstruction fidelity.
For instance, Zhicui Zhou et al. introduced a dual-channel upsampling module in Remote Sensing Image Road Segmentation Method Integrating CNN-Transformer and UNet. By merging pixel shuffle and bilinear interpolation, their decoding strategy balances the high-resolution reconstitution of pixel shuffling with the smoothing effects of interpolation, effectively eliminating fragmented boundaries and checkerboard patterns.
A similar hybrid strategy was proposed by Guorui Feng et al. in A Fusion Deraining Network Based on Swin Transformer and Convolutional Neural Network, where combining bilinear interpolation with pixel shuffling minimized information loss and yielded sharper final outputs compared to standard transposed convolutions.