Unsharp Mask Guided Filtering

Zenglin Shi, Yunlu Chen, Efstratios Gavves, Pascal Mettes, Cees G. M. Snoek

TL;DR

This paper introduces a novel guided image filtering approach inspired by unsharp masking, simplifying the process by estimating a single coefficient instead of two. This leads to improved structure transfer and fewer artifacts, and a new network architecture allows for a flexible trade-off between accuracy and efficiency.

What Problem Does This Paper Address?

Image filtering is crucial for tasks like noise reduction and detail enhancement. While traditional filters are content-blind, guided filters leverage a separate guidance image to transfer structural information, preserving important features like edges. However, existing deep learning-based guided filters often struggle with halo artifacts and inconsistent structure transfer, partly due to estimating two core coefficients simultaneously. This paper aims to address these limitations by proposing a more robust and efficient guided filtering formulation.

Methods

The core innovation is a new guided filtering formulation inspired by unsharp masking. Instead of estimating two coefficients (akin to the original guided filter), this method estimates a single coefficient that controls the amount of structure transferred from the guidance image. This is achieved by first creating "unsharp masks" of both the target and guidance images using a low-pass filter, and then using these masks as input to a neural network that predicts the single coefficient. This coefficient then dictates how the guidance image's structure is blended with a smoothed version of the target image.

To enable a trade-off between accuracy and speed, the authors introduce a "successive guided filtering network." This network is designed as a series of blocks, where each block can produce a filtering output. This allows users to choose a result from an earlier block for faster processing or a result from a later block for higher accuracy, all from a single trained network. The network utilizes dilated convolutions to effectively capture multi-scale structural information.

Datasets

The paper evaluates its method on several standard datasets:

  • NYU Depth V2: Used for depth upsampling and denoising.
  • Sintel: Used for optical flow upsampling.
  • BSDS500: Used for natural image denoising.
  • Flash/no-flash dataset: Used for cross-modality denoising.
  • RGB/NIR dataset: Used for cross-modality denoising.
  • bw/color dataset: Used for cross-modality joint upsampling.
  • RGB/saliency dataset: Used for cross-modality joint upsampling.

Key Insights and Ideas

  • Unsharp Masking Inspiration: Applying the principle of unsharp masking (enhancing details by adding a high-pass filtered version of the image to the original) to guided filtering provides a more intuitive and effective way to control structure transfer.
  • Single Coefficient Sufficiency: Estimating a single coefficient for structure transfer is sufficient and leads to better results than estimating two, reducing artifacts like halos.
  • Unsharp Masks as Input: Using unsharp masks of the target and guidance images as input to the network, rather than the raw images, focuses the learning on structural information and improves convergence.
  • Successive Filtering for Trade-offs: The proposed network architecture allows for a flexible trade-off between filtering accuracy and computational efficiency by enabling the selection of outputs from different stages of the network.
  • Explicit Structure Transfer: The method emphasizes explicit structure transfer, which is more robust than implicit methods that rely solely on feature fusion.

Key Results

  • State-of-the-Art Performance: The proposed method achieves state-of-the-art results across various tasks, including depth upsampling (e.g., 6.07 RMSE on NYU Depth V2 for 16x upsampling), denoising (e.g., best PSNR/SSIM on BSDS500 and NYU Depth V2), and cross-modality filtering.
  • Reduced Artifacts: Qualitatively, the method significantly reduces halo artifacts and preserves finer edges compared to existing methods like GF, DGF, DJF, and SVLRM.
  • Efficiency: The successive filtering network allows for a range of performance-accuracy trade-offs. For instance, the fastest output (Î(1)) achieves competitive run-time (31 ms) while still outperforming some baselines.
  • Robustness: The method is robust to different smoothing filter types and sizes used for generating unsharp masks.

Limitations and Future Work

  • Hyperparameter Tuning: While the paper demonstrates robustness, optimal performance still relies on selecting appropriate hyperparameters for the smoothing filter and network architecture.
  • Successive Filtering Depth: The paper notes that performance gains from increasing the number of successive filtering blocks diminish after a certain point (e.g., L=4 or L=5), suggesting that deeper networks might not always yield proportional improvements.
  • Future Work: The authors suggest exploring more advanced network architectures and potentially extending the formulation to other image processing tasks. The code is made publicly available, encouraging further research and application.

Summary updated

Abstract

The goal of this paper is guided image filtering, which emphasizes the importance of structure transfer during filtering by means of an additional guidance image. Where classical guided filters transfer structures using hand-designed functions, recent guided filters have been considerably advanced through parametric learning of deep networks. The state-of-the-art leverages deep networks to estimate the two core coefficients of the guided filter. In this work, we posit that simultaneously estimating both coefficients is suboptimal, resulting in halo artifacts and structure inconsistencies. Inspired by unsharp masking, a classical technique for edge enhancement that requires only a single coefficient, we propose a new and simplified formulation of the guided filter. Our formulation enjoys a filtering prior from a low-pass filter and enables explicit structure transfer by estimating a single coefficient. Based on our proposed formulation, we introduce a successive guided filtering network, which provides multiple filtering results from a single network, allowing for a trade-off between accuracy and efficiency. Extensive ablations, comparisons and analysis show the effectiveness and efficiency of our formulation and network, resulting in state-of-the-art results across filtering tasks like upsampling, denoising, and cross-modality filtering. Code is available at https://github.com/shizenglin/Unsharp-Mask-Guided-Filtering.
BibTeX
@article{shi2021unsharp,
  author = {Zenglin Shi and Yunlu Chen and Efstratios Gavves and Pascal Mettes and Cees G. M. Snoek},
  title = {{Unsharp Mask Guided Filtering}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=4kHsd4-vtR8},
}

Loading related content...

Figure