In the design of convolutional neural networks, downsampling is often treated as a necessary evil—a way to reduce computational load and increase receptive fields. However, the standard tools for this task, such as average pooling or strided convolutions, act as low-pass filters. They smooth out signals, effectively discarding the high-frequency details that define edges, textures, and tiny structures.
A growing body of research suggests that by replacing "averaging" with "selection"—specifically through max or min pooling—networks can better preserve the fragile, high-frequency information critical for tasks ranging from satellite imaging to depth estimation.
The Low-Pass Filter Problem
When a network averages a block of pixels, it performs a mathematical "blur." This is acceptable for identifying large, homogenous objects like a blue sky or a green field. But for tasks that rely on sharp transitions, averaging is destructive.
In the context of pan-sharpening—the process of fusing low-resolution color images with high-resolution grayscale images—Huanyu Zhou, Yunhong Wang, and colleagues argue that average pooling is a primary culprit for blurry outputs. In their work on Unsupervised Cycle-consistent Generative Adversarial Networks for Pan-sharpening, they demonstrate that using maximum pooling in spatial reconstruction loss functions prevents the blurring of edges. By selecting the most prominent local activation rather than an average, the network maintains the sharp boundaries found in high-resolution satellite imagery.
Max Pooling as a Frequency Selector
The intuition behind using max pooling for signal preservation is that high-frequency details (like a sharp edge) manifest as high-magnitude activations in feature maps. While an average operation would dilute these peaks with the surrounding low-value background, max pooling "picks" the peak and carries it forward.
This selective mechanism is particularly vital in hybrid architectures. Chenyang Si, Weihao Yu, and their team noted in the Inception Transformer that standard Vision Transformers (ViTs) tend to behave as low-pass filters. To counteract this, they integrated max pooling into their "Inception mixer." They found that max pooling is uniquely sensitive to sharp local changes, allowing the model to capture fine-grained textures that self-attention layers often smooth over.
Similarly, in the field of Depth from Focus (DfF), the goal is to determine distance by identifying which parts of an image are perfectly sharp. Changyeon Won and Hae-Gon Jeon show in Learning Depth from Focus in the Wild that max pooling is a superior downsampling strategy compared to strided convolutions. Because focus cues are inherently high-frequency signals, max pooling acts as a natural filter that selects for the sharpest activations, ensuring that the "infocus" signal isn't lost as the data moves deeper into the network.
Preserving the Weakest Signals
While max pooling preserves the "loudest" high-frequency signals, some tasks require the opposite: preserving the "faintest" signals. In Pixel-Semantic Revising of Position, Qian Li and colleagues explore the challenge of detecting tiny objects against complex backgrounds.
They propose that for extremely small objects, minimum pooling can be more effective than max or average pooling. The intuition here is that tiny structures might be represented by subtle, low-intensity features that are easily overwhelmed by dominant background noise. By using minimum pooling, the network can preserve these "weak" signals that would otherwise be smoothed over or out-competed by the "maximum" activations of the background.
Why It Works
The shift toward selective pooling (Max/Min) across these diverse papers reveals a shared principle: Downsampling should be a feature-selection process, not a signal-reduction process.
- Contrast Preservation: Unlike averaging, which reduces the dynamic range of a feature map, selective pooling maintains the contrast between high and low activations.
- Aliasing Mitigation: By picking the most representative signal (the sharpest edge or the smallest detail), these methods avoid the "ghosting" or blurring artifacts that occur when disparate signals are mathematically blended.
- Task-Specific Filtering: Max pooling acts as a high-pass filter for edges and focus cues, while min pooling can act as a filter for subtle foreground signals against noisy backgrounds.
By treating pooling as a strategic choice rather than a default operation, researchers like Huanyu Zhou and Chenyang Si have shown that deep networks can maintain high-fidelity spatial awareness even as they scale to lower resolutions.