Spatially Varying Dynamic Convolution

23 concepts19 papers

Standard convolutional neural networks (CNNs) owe their success to two core principles: weight sharing and translation equivariance. By sliding the exact same filter across an entire image, a CNN can efficiently detect features regardless of where they appear.

However, this "one-size-fits-all" assumption breaks down when dealing with complex, non-uniform spatial phenomena. Consider an image obscured by motion blur: the direction and magnitude of the blur vary depending on camera movement and object depth. Similarly, in tasks like image inpainting or adverse weather removal, some regions are completely clean while others are severely degraded. Applying a globally uniform filter across these distinct regions forces the network to compromise, often resulting in blurry edges or artifacts.

To solve this, a powerful paradigm has emerged: spatially varying dynamic convolution (also known as per-pixel dynamic filtering). Instead of learning a fixed set of static weights during training, the network dynamically predicts a unique convolutional kernel for every single pixel coordinate at runtime, conditioned on the local and global context of the input.


The Core Mechanism: Kernel Prediction Networks

The fundamental architecture of a spatially varying dynamic convolution layer splits the processing into two distinct pathways: a kernel prediction branch and a filtering branch.

Input Feature Map ──┬──> Kernel Prediction Network ──> Predicted Kernels K(x,y)
                    │                                         │
                    └─────────────────────────────────────────▼──> Dynamic Filtering Layer ──> Output
  1. Kernel Prediction: A lightweight sub-network processes the input feature map and outputs a dense tensor of coefficients. If the target layer uses a kernel size of $k \times k$, the prediction network generates a tensor of shape $H \times W \times k^2$ (or $H \times W \times C \times k^2$ if the filters are channel-specific).
  2. Dynamic Filtering: For each pixel coordinate $(x, y)$, the corresponding predicted weights are reshaped into a $k \times k$ matrix. This unique kernel is then convolved with the local patch centered at $(x, y)$ in the input feature map.

By decoupling the filter weights from the network's static parameters, the model transforms from a rigid feature extractor into a flexible, content-adaptive processor. As Sim et al. demonstrate in their work on motion deblurring, this allows the network to perform localized spatial transformations tailored precisely to the fine details and textures of each neighborhood.


To understand why this approach is so potent, it helps to contrast it with self-attention mechanisms found in Transformers. Standard self-attention computes pairwise relationships between all pixels, resulting in a quadratic computational complexity ($O(N^2)$) that becomes prohibitive for high-resolution images.

In Involution, Li et al. formalize an elegant alternative. They propose "inverting" the design principles of convolution by making the operator spatial-specific (different weights for different locations) but channel-agnostic (sharing weights across channels).

The authors argue that the primary benefit of self-attention in vision tasks is not necessarily the long-range relational modeling, but rather its spatial adaptivity. By generating localized, pixel-specific kernels from a pixel's own features, dynamic filtering achieves the same content-adaptive representation power as attention, but within a localized receptive field and without the quadratic complexity of pairwise comparisons.


Adapting to Non-Uniform Degradations

The most immediate application of per-pixel dynamic filtering is in low-level image restoration, where degradations are inherently spatially variant.

Motion and Video Deblurring

In video and dynamic scenes, blur is rarely uniform. Zhou et al. introduced the Spatio-Temporal Filter Adaptive Network (STFAN), which utilizes a Filter Adaptive Convolutional layer to handle both frame alignment and deblurring simultaneously in the feature domain. Instead of relying on error-prone explicit optical flow, the spatially-varying filters dynamically align features across frames based on local context.

To expand the effective receptive field without adding massive computational depth, Suin et al. developed a network (SAPHN) that pairs pixel-dependent filters with dynamic spatial offsets. This allows the kernel to adapt its shape and footprint dynamically, gathering scattered information across non-uniform blur trajectories.

Variational Noise, Weather, and Highlights

Real-world restoration requires models to handle multiple, unpredictable degradation levels within a single frame. Xu et al. addressed this in their Unified Dynamic Convolutional Network (UDVD) for super-resolution, where per-pixel kernels adapt their restoration logic depending on local blur and noise estimates.

This selective targeting is also vital for adverse weather removal. As explored in recent pre-trained model applications, spatially-adaptive dynamic convolutions prevent the network from blurring clean regions of an image while aggressively targeting pixels obscured by localized raindrops or snow streaks.

For highly localized distortions like specular highlights on text documents, Jiang demonstrated in HAFNet that predicting unique filtering kernels allows the network to adjust its filtering strategy based on the intensity of the highlight, preventing the "black hole" artifacts common to static filtering methods.


Structural Adaptivity and Context Fusion

Beyond removing noise, per-pixel filters excel at tasks requiring precise structural boundaries or the blending of disparate data sources.

  • Geometry-Aware Segmentation: In retinal vessel segmentation, Li et al. utilize pixel-wise adaptive filters to capture complex local geometric structures. Because the kernels adapt to the immediate neighborhood, they can trace thin boundaries and complex vascular junctions that standard convolutions typically smooth over.
  • Image Inpainting: Guo et al. reframed image inpainting as a dynamic filtering problem (MISF). By predicting content-adaptive kernels on-the-fly, the model generalizes across diverse hole shapes and structural patterns, ensuring local consistency.
  • Multi-Modal Fusion: When merging different image modalities (such as panchromatic and multispectral images in remote sensing), Peng et al. developed ADKNet. Their network generates source-adaptive discriminative kernels that treat spatial details and spectral information distinctly at each pixel coordinate. Similarly, the Mutual-Guided Dynamic Network (MGDN) by Guan et al. uses dynamic filters to apply different operations to focused versus defocused areas during image fusion.
  • Global Consistency: While local adaptivity is crucial, some tasks require global harmony. Zhang et al. designed GKNet for image harmonization, generating unique kernels for every pixel by integrating local features with a Transformer-based global context extractor. This ensures that composited foreground objects match their backgrounds with both local precision and global environmental consistency.

Overcoming the Computational Bottleneck

The primary challenge of per-pixel dynamic convolution is its heavy memory and computational footprint. Generating, storing, and applying a unique $k \times k$ kernel for every single pixel can quickly bottleneck hardware.

To mitigate this, researchers have introduced several efficiency innovations:

  • Lightweight MLPs: In DSTNet, Pan et al. generate dynamic spatial filters via highly optimized, lightweight multi-layer perceptrons. This achieves high representational power and content-adaptivity without the massive overhead of standard self-attention mechanisms.
  • Malleable Convolution: Jiang et al. proposed Malleable Convolution (MalleConv). Instead of predicting kernels at the full image resolution, MalleConv predicts kernels on a downsampled grid. It then applies these weights across local patches using a memory-efficient "on-the-fly slicing" operator, providing the flexibility of per-pixel filtering at a fraction of the computational cost.
Full Resolution Image ──> Downsample ──> Predict Kernels on Grid ──> Slice & Interpolate ──> Efficient Application
  • Locally Masked Inputs: Rather than dynamically changing the weights themselves, Jain et al. introduced LMCONV, which applies location-specific binary masks to input patches while keeping the convolutional weights static. This allows the causal neighborhood to vary across coordinates, enabling flexible autoregressive generation in arbitrary orders without the overhead of full weight generation.
  • Mathematical Constraints: He et al. developed Spatially Adaptive Equivariant Partial Differential Operators (Neural ePDOs). Instead of predicting arbitrary kernel coefficients, a small network generates position-specific coefficients for structured differential operators. This maintains translation equivariance and structural discipline, allowing the model to achieve state-of-the-art results with a fraction of the parameters.

By shifting from static weight maps to context-driven, coordinate-specific kernel generation, spatially varying dynamic convolution bridges the gap between the efficiency of standard CNNs and the content-adaptivity of Transformers, providing a precise and scalable tool for complex visual understanding and restoration.

Key Foundational Work

Unified Dynamic Convolutional Network for Super-Resolution With Variational Degradations (2020) — contributed 2 concept elements to this direction.
Spatially-Attentive Patch-Hierarchical Network for Adaptive Motion Deblurring (2020) — contributed 2 concept elements to this direction.
A Deep Motion Deblurring Network Based on Per-Pixel Adaptive Kernels With Residual Down-Up and Up-Down Modules (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_per_pixel_dynamic_filtering_4d0d6866,
  title = {per-pixel dynamic filtering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/per-pixel-dynamic-filtering/rdr_4d0d686604354483ac91f64090a5ed77},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_per_pixel_dynamic_filtering_4d0d6866_v1,
  title = {per-pixel dynamic filtering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/per-pixel-dynamic-filtering/rdr_4d0d686604354483ac91f64090a5ed77/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}

See something wrong? Report a mistake