Lossless Spatial Downsampling via Pixel Unshuffling

29 concepts22 papers1 proposal

In convolutional neural networks, downsampling is an essential tool for managing computational complexity and expanding the receptive field. Traditionally, networks achieve this by using max pooling, average pooling, or strided convolutions. While these operations successfully reduce the spatial dimensions of a feature map, they do so by discarding information. Taking the maximum or average value of a $2 \times 2$ neighborhood inherently obliterates high-frequency details, fine textures, and sub-pixel precision.

As noted by Joern-Henrik Jacobsen et al. in their work on i-RevNet: Deep Invertible Networks, standard architectures operate on an information bottleneck principle, progressively throwing away "noise" to extract categorical essence. However, for low-level vision and high-fidelity reconstruction tasks, this discarded "noise" is precisely the signal we care about.

To resolve this, a parallel paradigm has emerged: lossless spatial downsampling, often implemented via pixel unshuffling or spatial-to-channel rearrangement. Instead of mathematically compressing a neighborhood of pixels into a single value, this technique reallocates spatial pixels directly into the channel dimension.


The Mechanics of Pixel Unshuffling

To understand how pixel unshuffling works, imagine a single-channel image of size $4 \times 4$. We want to downsample this image by a factor of 2, resulting in a spatial resolution of $2 \times 2$.

Instead of averaging adjacent pixels, we divide the image into a grid of $2 \times 2$ blocks. Each block contains 4 distinct pixels. We then peel these pixels apart based on their relative coordinates within the block (e.g., top-left, top-right, bottom-left, bottom-right) and stack them along the channel axis.

Input (4x4x1)                  Output (2x2x4)
+---+---+---+---+              +---+---+  +---+---+  +---+---+  +---+---+
| A | B | A | B |              | A | A |  | B | B |  | C | C |  | D | D |
+---+---+---+---+              +---+---+  +---+---+  +---+---+  +---+---+
| C | D | C | D |  =======>    | A | A |  | B | B |  | C | C |  | D | D |
+---+---+---+---+  Unshuffle   +---+---+  +---+---+  +---+---+  +---+---+
| A | B | A | B |              Channel 1  Channel 2  Channel 3  Channel 4
+---+---+---+---+              (Top-Left) (Top-Right)(Bot-Left) (Bot-Right)
| C | D | C | D |
+---+---+---+---+

Mathematically, a tensor of shape $(H, W, C)$ is reshaped into a tensor of shape:

$$\left(\frac{H}{r}, \frac{W}{r}, r^2 \cdot C\right)$$

where $r$ is the downsampling factor.

Because this operation is a pure geometric rearrangement of data, it is entirely bijective and invertible. No mathematical averaging occurs, meaning zero information is lost. This mechanism has been described across the literature under various names, including "spatial-to-channel squeezing" by Zhenyue Qin et al., "neighborhood pixel packing" by Gen Luo et al., and "pixel sampling and channel stacking" by Hongbing Lyu et al..


Why It Works: Preserving High-Frequency Details

The primary benefit of pixel unshuffling is that it allows a neural network to enjoy the computational efficiency of operating at a lower spatial resolution without suffering from the information bottlenecks of traditional downsampling.

In dense prediction and image restoration tasks, maintaining high-frequency spatial precision is critical:

  • Denoising and Deblurring: In Self-Guided Network for Fast Image Denoising, Shuhang Gu et al. demonstrate that shuffling spatial pixels into channels preserves the fine-grained textures necessary to recover clean images from noise. Similarly, Kiyeon Kim et al. utilize pixel unshuffling in MSSNet: Multi-Scale-Stage Network for Single Image Deblurring to preserve data density when processing images at coarser scales.
  • Super-Resolution: Real-world super-resolution challenges, such as those detailed by Jianrui Cai et al. in the NTIRE 2019 Challenge on Real Image Super-Resolution, highlight pixel unshuffling as a key mechanism to avoid losing sub-pixel precision before upsampling. Bin Sun et al. leverage this in their Hybrid Pixel-Unshuffled Network for Lightweight Image Super-resolution to achieve lightweight yet highly accurate reconstructions.
  • Matting and Segmentation: For tasks like portrait matting, preserving fine boundaries like hair or fur is notoriously difficult. Peng Liu et al. show that replacing standard bilinear interpolation with channel-reorganizing downsampling prevents the mathematical blurring of edges. Yatao Zhong similarly uses pixel-unshuffle tokenization to retain full signal integrity during high-resolution portrait processing.
  • Spectral Reconstruction: When reconstructing high-dimensional hyperspectral data from standard RGB inputs, Yuzhi Zhao and Beibei Song et al. independently demonstrate that lossless spatial rearrangement prevents critical spatial-spectral details from being bottlenecked.

By shifting the spatial dimensions into the channel dimensions, subsequent convolutional layers can learn how to combine these sub-pixel elements via standard channel-mixing operations, effectively letting the network learn how to downsample based on the task context.


Breaking the Memory Wall via Invertibility

Beyond preserving visual features, lossless downsampling plays a vital role in optimizing network training memory through reversible neural networks (RevNets).

During standard backpropagation, a network must store all intermediate activations in memory to calculate gradients. This creates a massive memory bottleneck, especially for high-resolution inputs. Reversible networks circumvent this by recomputing activations on the fly during the backward pass, eliminating the need to store them.

However, for a network to be fully reversible, every single layer—including the downsampling layers—must be mathematically invertible.

Forward Pass:  Input (H, W, C)   =======>  Pixel Unshuffle  =======>  Output (H/2, W/2, 4C)
Backward Pass: Input (H, W, C)  <=======   Pixel Shuffle   <=======  Output (H/2, W/2, 4C)

Because traditional pooling and strided convolutions discard data, they are non-invertible. Bijective tensor reshaping solves this problem perfectly. As shown by Bei Liu et al. in speaker verification and Vitaliy Chiley et al. in RevBiFPN: The Fully Reversible Bidirectional Feature Pyramid Network, integrating invertible spatial-to-channel downsampling allows networks to maintain a fully mathematically traceable path from the deepest representations back to the original input.

This architectural choice enables massive memory savings (often up to 15x to 20x reductions), allowing deep, high-resolution models to be trained on standard hardware constraints without sacrificing accuracy.

Key Foundational Work

Self-Guided Network for Fast Image Denoising (2019) — contributed 2 concept elements to this direction.
i-RevNet: Deep Invertible Networks (2018) — contributed 1 concept elements to this direction.
NTIRE 2019 Challenge on Real Image Super-Resolution: Methods and Results (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_information_preserving_downsampling_1b21ffbf,
  title = {information-preserving downsampling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/information-preserving-downsampling/rdr_1b21ffbf095e4e2a8f31f132365087c6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_information_preserving_downsampling_1b21ffbf_v1,
  title = {information-preserving downsampling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/information-preserving-downsampling/rdr_1b21ffbf095e4e2a8f31f132365087c6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake