LiftPool: Bidirectional ConvNet Pooling

Jiaojiao Zhao, Cees G. M. Snoek

TL;DR

LiftPool is a novel bidirectional pooling strategy that replaces lossy operations like MaxPool with an invertible "Lifting Scheme" from signal processing. By decomposing feature maps into multiple frequency sub-bands, it preserves fine-grained details during downsampling and enables high-quality, refined reconstruction during upsampling. This leads to better accuracy and significantly improved robustness to noise.

The Problem with Traditional Pooling

In Convolutional Neural Networks (ConvNets), pooling is essential. It increases the "receptive field" (allowing the network to see the big picture) and makes the model robust to small translations. However, traditional pooling methods—like MaxPool or AveragePool—are inherently "lossy" processes.

When an image is downsampled by picking the maximum value in a 2x2 window, 75% of the data is discarded. This creates two major headaches:

  1. Non-invertibility: In tasks like semantic segmentation where the image must be scaled back up to its original size, it is impossible to perfectly recover what was lost. Upsampling often results in "holes" or sparse, blocky feature maps.
  2. Aliasing: Simple downsampling ignores the high-frequency structures of the signal, frequently introducing visual artifacts (aliasing).
Comparison of MaxPool and LiftPool

The authors of LiftPool ask: Can we design a pooling layer that downsizes feature maps without losing the underlying information?

Methods: The Lifting Scheme

LiftPool draws inspiration from the Lifting Scheme, a classic technique in signal processing used for wavelets. The core idea is to decompose a signal into an approximation (low-frequency) and details (high-frequency).

LiftPool Architecture

The process works in three learnable steps:

  1. Split ($\mathcal{S}$): The input signal is split into even and odd indexed samples.
  2. Predict ($\mathcal{P}$): A small neural network predicts the odd samples from the even ones. The difference between the actual odd sample and the prediction becomes the detail sub-band ($d$).
  3. Update ($\mathcal{U}$): Another small network uses these details to "smooth" the even samples, creating a refined approximation ($s$).

For 2D images, this process is applied horizontally and then vertically, resulting in four sub-bands: LL (low-frequency approximation), and LH, HL, HH (details in horizontal, vertical, and diagonal directions).

  • LiftDownPool: For classification, the network can use the LL band or a combination of all sub-bands. Because the details are preserved rather than discarded, the network retains more discriminative power.
  • LiftUpPool: Because the lifting scheme is mathematically invertible, it can be run backward. By using the stored "details" from the downsampling phase, LiftUpPool reconstructs a dense, refined high-resolution feature map, avoiding the sparse look of traditional up-pooling.

Key Insights

  • Invertibility as a Feature: Most pooling layers are a one-way street. LiftPool is a two-way bridge. This is particularly powerful for "Encoder-Decoder" architectures (like U-Nets), where the decoder can now "reach back" and use the exact details filtered out during encoding.
  • Learnable Anti-aliasing: Unlike "BlurPool," which uses a fixed filter to reduce aliasing, LiftPool learns how to smooth the signal via the Update ($\mathcal{U}$) function. This allows the pooling layer to adapt to the specific features of the dataset.
  • Vertical Details Matter: Ablation studies showed that vertical details (the HL sub-band) often contributed more to classification accuracy than horizontal ones, suggesting that pooling isn't just about downsizing—it's about feature selection.

Experimental Results

The authors evaluated LiftPool across several standard benchmarks, including ImageNet for classification, PASCAL-VOC12 for segmentation, and ImageNet-C/P for robustness.

  • Better Accuracy: On ImageNet with a ResNet50 backbone, LiftDownPool achieved a 22.36% Top-1 error, significantly outperforming standard MaxPool (24.26%).
  • Superior Segmentation: In semantic segmentation tasks using SegNet, switching to LiftUpPool boosted the Mean Intersection over Union (mIoU) from 62.7 to 68.9. The resulting segmentation maps have much smoother edges and fewer artifacts.
  • Rock-Solid Robustness: LiftPool showed much higher stability against input corruptions. On the ImageNet-C benchmark, it consistently yielded lower error rates across various noise and blur types compared to all other pooling methods. It also improved "shift robustness," meaning the network is less likely to change its prediction if an image is shifted by just one pixel.

Limitations and Future Work

While LiftPool is powerful, the paper notes a few areas for future exploration:

  • Symmetry: Currently, the horizontal and vertical dimensions are treated equally. Future work could explore treating these dimensions differently based on image content.
  • Complexity: While the compute cost is minimal when summing sub-bands, the addition of learnable $\mathcal{P}$ and $\mathcal{U}$ functions does add parameters to the model compared to parameter-free operations like MaxPool.
  • Task Expansion: The authors suggest LiftPool could be highly effective for other detail-heavy tasks like image super-resolution or colorization.

Summary updated

Abstract

Pooling is a critical operation in convolutional neural networks for increasing receptive fields and improving robustness to input variations. Most existing pooling operations downsample the feature maps, which is a lossy process. Moreover, they are not invertible: upsampling a downscaled feature map can not recover the lost information in the downsampling. By adopting the philosophy of the classical Lifting Scheme from signal processing, we propose LiftPool for bidirectional pooling layers, including LiftDownPool and LiftUpPool. LiftDownPool decomposes a feature map into various downsized sub-bands, each of which contains information with different frequencies. As the pooling function in LiftDownPool is perfectly invertible, by performing LiftDownPool backward, a corresponding up-pooling layer LiftUpPool is able to generate a refined upsampled feature map using the detail subbands, which is useful for image-to-image translation challenges. Experiments show the proposed methods achieve better results on image classification and semantic segmentation, using various backbones. Moreover, LiftDownPool offers better robustness to input corruptions and perturbations.
BibTeX
@inproceedings{zhao2021liftpool,
  author = {Jiaojiao Zhao and Cees G. M. Snoek},
  title = {{LiftPool: Bidirectional ConvNet Pooling}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=meUX0FB-Abx},
}
Figure