Deep learning models excel at understanding global context and complex semantics, but running them directly on multi-megapixel images is computationally prohibitive. High-resolution convolutional layers consume massive amounts of memory and processing time.
The intuitive workaround is straightforward: downsample the input image, run the deep network at a lower resolution, and then upsample the network's output back to the original scale. However, standard upsampling methods like bilinear or bicubic interpolation introduce severe artifacts. They blur sharp boundaries, cause color bleeding, and create halos. This happens because standard spatial interpolation is content-agnostic; it assumes smooth transitions everywhere, completely ignoring the true underlying geometry of the scene.
Bilateral guided upsampling solves this dilemma by decoupling the computation of content from the preservation of structure.
The Core Insight: Decoupling Content and Structure
Why can we get away with low-resolution processing in the first place? In many vision tasks—such as estimating illumination maps, disparity fields, color channels, or semantic features—the underlying signal varies smoothly over large regions. The high-frequency details (the exact locations of edges and boundaries) are already perfectly preserved in the original high-resolution input image.
Instead of forcing a deep network to output a high-resolution map directly, we can use the original high-resolution image as a guidance map. By combining the low-resolution deep features with the high-resolution guide, an edge-preserving filter can upscale the features while snapping their boundaries to the true edges of the scene.
Low-Res Input ──► [ Deep Neural Network ] ──► Low-Res Output Map ──┐
▼
High-Res Guide ──────────────────────────────────────────────► [ Guided Upsampling ] ──► High-Res Output
The Bilateral Grid and Joint Bilateral Filtering
To understand why this works, consider a standard bilateral filter. When smoothing an image, it weights neighboring pixels based on two metrics: spatial distance (how close pixels are in space) and range distance (how similar they are in color or intensity).
In joint bilateral upsampling, the range distance is computed from a separate guidance image rather than the low-resolution map being filtered. When upscaling a low-resolution pixel, the operator looks at the corresponding local neighborhood in the high-resolution guide. If two adjacent high-resolution pixels have very different colors, they likely belong to different objects. The filter assigns a low weight to transitions across this boundary, preventing the low-resolution values from mixing.
To make this operation fast enough for real-time applications, researchers project the low-resolution features into a low-dimensional bilateral grid—typically a 3D or 4D space spanning spatial coordinates and intensity/color. Upsampling then simplifies into a fast slicing operation guided by the high-resolution image.
Standard Bilateral Weight: W = Spatial_Kernel(Δx) × Range_Kernel(ΔI_target)
Joint Bilateral Weight: W = Spatial_Kernel(Δx) × Range_Kernel(ΔI_guidance)
This mechanism is widely used in image enhancement. Jiafei Wu et al. and Bo Liu apply it to estimate high-resolution illumination maps for underexposed photos. By predicting the illumination at a low resolution and slicing through a bilateral grid, they achieve real-time enhancement without halo artifacts. Similarly, Mahmoud Afifi et al. leverage bilateral guided upsampling within a multi-scale Laplacian pyramid framework to correct photo exposure with minimal memory overhead, while K. Ram Prabhakar et al. use it to preserve fine structural details during high-resolution HDR deghosting.
This approach is equally powerful for geometric tasks. Bin Xu et al. perform heavy cost aggregation for stereo matching at a low resolution, then use a learned high-resolution guidance map to slice a 4D bilateral grid via trilinear interpolation. This yields real-time disparity maps that perfectly respect fine textures and sharp object boundaries. For image smoothing, Qing Zhang et al. employ joint bilateral filtering at each stage of a Gaussian-Laplacian pyramid, injecting high-frequency structural edges into the upsampled result without reintroducing unwanted textures.
Guided Filtering and Content-Adaptive Convolutions
Another facet of this research direction replaces explicit bilateral filtering with learned, content-adaptive operators. A guided filter assumes a local linear relationship between the guidance image and the output, allowing structural details to transfer seamlessly across scales via linear coefficients.
Output_pixel = a × Guidance_pixel + b
(where coefficients 'a' and 'b' are computed locally)
The authors of Attention Guided Network for Retinal Image Segmentation integrate these edge-preserving guided filters directly into the decoder path of a convolutional network, ensuring fine blood vessels do not blur out during segmentation. For reflection removal, B. H. Pawan Prasad et al. utilize convolutional guided filters to progressively refine feature maps across resolution scales on mobile devices.
Taking this concept further, Hang Su et al. propose Pixel-Adaptive Convolutions (PAC). In a PAC layer, standard filter weights are dynamically multiplied by a spatially varying kernel that depends on high-resolution guidance features:
$$\text{PAC}(F)_{i} = \sum_{j \in \Omega(i)} K(v_i, v_j) \cdot W_{i-j} \cdot F_j$$
Where $F$ is the feature map, $W$ is the standard filter weight, and $K$ is a kernel function measuring the similarity of guidance features $v$ at pixels $i$ and $j$. This generalizes bilateral filtering into a fully differentiable network layer, preventing the blurring of boundaries in joint upsampling tasks like depth and optical flow reconstruction.
Eliminating Artifacts Across Diverse Domains
The versatility of edge-preserving guided upsampling is evident across a wide range of applications where resolution mismatches occur:
- Colorization and Fusion: In image colorization, Jiaojiao Zhao et al. use a high-resolution grayscale input as a guide to upscale low-resolution color maps, completely eliminating "edge color bleeding." For multi-exposure image fusion, Ting Jiang 0005 et al. estimate fusion weights at a reduced resolution and restore them via guided filtering to keep weight transitions aligned with structural boundaries.
- View Synthesis: When synthesizing novel viewpoints from visually imbalanced stereo pairs, Yicun Liu et al. employ a deep guided filtering layer to keep synthesized edges sharp. In a single-image setting, Yicun Liu demonstrates that using the original input image as a structural guide for elastic displacement maps prevents foreground objects from bleeding into backgrounds.
- Feature Refinement and Super-Resolution: The framework FeatUp applies a stack of learned bilateral filters to upscale low-resolution features from any vision backbone (like ViTs or CLIP), ensuring feature boundaries align precisely with object edges. Even in ultra-efficient super-resolution, Jinjin Gu et al. use a blending mask derived from predicted offsets to seamlessly transition between cheap interpolation in flat regions and neural refinement along high-contrast edges.
By shifting the heavy lifting of deep networks to a lower spatial scale and using the high-resolution input to guide the final upsampling step, these methods achieve the best of both worlds: the semantic power of deep learning and the crisp, edge-accurate fidelity of high-resolution imagery.
Key Foundational Work
Underexposed Photo Enhancement Using Deep Illumination Estimation (2019) — contributed 2 concept elements to this direction.
Pixel-level Semantics Guided Image Colorization (2018) — contributed 2 concept elements to this direction.
Double Refinement Network for Efficient Monocular Depth Estimation (2019) — contributed 1 concept elements to this direction.