Robust Guided Image Filtering Using Nonconvex Potentials

Bumsub Ham, Minsu Cho, Jean Ponce

TL;DR

This paper introduces the "SD filter," a novel approach to guided image filtering that robustly handles structural differences between input and guidance images and is resilient to outliers. By formulating guided filtering as a nonconvex optimization problem solved with majorize-minimization, the SD filter effectively leverages both static (e.g., color) and dynamic (e.g., filtered input) guidance signals, leading to superior performance across various computer vision tasks.

What Problem Does This Paper Address?

Traditional guided image filtering methods often struggle with two key issues:

  1. Structural Mismatches: They assume the input and guidance images have similar structures. When this isn't the case (e.g., using a color image to guide a depth image from a different sensor), these methods can transfer unwanted structures or smooth over important details.
  2. Robustness to Outliers: They are not inherently robust to noisy data or outliers, which can lead to artifacts like gradient reversal or global intensity shifts.

Existing methods typically rely on either static guidance (where the guidance signal is fixed) or dynamic guidance (where the guidance signal is updated iteratively). Static guidance can transfer too much information, while dynamic guidance might not fully exploit external structural priors. This paper aims to bridge this gap by creating a unified framework that effectively combines both static and dynamic guidance.

Methods

The core of the SD filter lies in its formulation as a nonconvex optimization problem. The objective function to be minimized consists of:

  • A fidelity term: Ensures the output image is consistent with the input image.
  • A regularization term: Promotes smoothness while preserving salient features.

The novelty lies in the regularization term, which uses a nonconvex potential (Welsch's function). This nonconvexity is key to achieving robustness to outliers and allowing the filter to better handle structural differences. The regularization term also incorporates:

  • Static guidance weighting: Uses a function ($\Phi_\mu$) that measures the similarity between adjacent pixels in the static guidance image. This transfers structural information from the guidance to the output.
  • Dynamic guidance weighting: Uses a function ($\Psi_\nu$) that penalizes large gradients in the output image (which acts as dynamic guidance). This ensures the output image itself has desirable structural properties and contributes to robustness.

This nonconvex optimization problem is solved using the majorize-minimization algorithm. This iterative approach involves:

  1. Majorization step: Constructing a convex surrogate function that bounds the original objective function.
  2. Minimization step: Finding the minimum of this surrogate function to update the output image.

The paper also proposes two initialization strategies for the majorize-minimization algorithm:

  • L2 initialization: Based on the Weighted Least Squares (WLS) framework, providing a convex initial estimate.
  • L1 initialization: Using an L1 norm regularizer, which is a tighter upper bound for Welsch's function and leads to faster convergence.

Datasets

The paper evaluates the SD filter on several standard datasets and applications:

  • Middlebury dataset [60]: Used for depth upsampling, a benchmark for stereo correspondence.
  • Graz dataset [10]: Also used for depth upsampling.
  • BSDS300 database [64]: Used for evaluating boundary localization in scale-space filtering.
  • Real-world datasets: Including those from structured light sensors and ToF cameras for depth upsampling.
  • Custom datasets: For texture removal, flash/non-flash denoising, and RGB/NIR denoising, often using images from the authors' project webpages or existing literature.

No new datasets were introduced; instead, the paper focuses on evaluating its method against existing state-of-the-art techniques on established benchmarks.

Key Insights and Ideas

  • The Power of Dual Guidance: The most significant insight is that combining both static and dynamic guidance is crucial for robust and accurate image filtering, especially when dealing with data from different sensors or with inherent structural mismatches. Static guidance provides external structural priors, while dynamic guidance leverages the evolving structure of the output itself.
  • Nonconvexity for Robustness: Using a nonconvex potential (Welsch's function) in the regularization term is key to achieving robustness against outliers and handling structural differences. This is a departure from many previous methods that relied on convex regularization.
  • Majorize-Minimization for Nonconvex Optimization: The paper demonstrates the effectiveness of the majorize-minimization algorithm for efficiently finding a local minimum of the nonconvex objective function, guaranteeing convergence.
  • Initialization Matters for Speed: While the algorithm converges to a local minimum regardless of initialization, using an L1-based initialization significantly speeds up convergence compared to L2-based initialization, without sacrificing accuracy.

Key Results

  • Superior Depth Upsampling: The SD filter consistently outperforms state-of-the-art methods (e.g., GF, TGV, WMF, WModF) in depth upsampling tasks, achieving lower Percentage of Bad Pixels (PBP) and better preservation of sharp depth edges and discontinuities, as evidenced by quantitative metrics and visual comparisons.
  • Effective Scale-Space Filtering and Texture Removal: The SD filter demonstrates strong performance in scale-space filtering, preserving object boundaries well even at coarse scales and being robust to global intensity shifts. For texture removal, it effectively removes textures while preserving important structures like corners.
  • Robustness to Artifacts: The method is shown to be robust to artifacts such as gradient reversal and global intensity shifts.
  • Efficiency: While solving a nonconvex optimization problem, the majorize-minimization algorithm converges quickly. For example, an L1 initialization can achieve convergence in as few as 7 steps, with processing times comparable to or better than other advanced methods.
  • Parameter Sensitivity: The L1 initialization shows less sensitivity to the regularization parameter ($\lambda$) compared to other methods, indicating a more stable performance.

Limitations and Future Work

  • Local Minima: The majorize-minimization algorithm guarantees convergence to a local minimum, not necessarily the global one. The choice of initialization can influence the final result.
  • Computational Cost: While efficient, the iterative nature of the algorithm still involves solving linear systems, which can be computationally intensive for very large images or real-time applications without further optimization.
  • Future Work: The authors suggest that their framework can be extended to improve refinement in end-to-end learning pipelines for tasks like semantic segmentation by incorporating the structures of both input and output CNN features. They also highlight the potential for applying the SD filter to other joint image restoration tasks.

Summary updated

Abstract

Filtering images using a guidance signal, a process called guided or joint image filtering, has been used in various tasks in computer vision and computational photography, particularly for noise reduction and joint upsampling. This uses an additional guidance signal as a structure prior, and transfers the structure of the guidance signal to an input image, restoring noisy or altered image structure. The main drawbacks of such a data-dependent framework are that it does not consider structural differences between guidance and input images, and that it is not robust to outliers. We propose a novel SD (for static/dynamic) filter to address these problems in a unified framework, and jointly leverage structural information from guidance and input images. Guided image filtering is formulated as a nonconvex optimization problem, which is solved by the majorize-minimization algorithm. The proposed algorithm converges quickly while guaranteeing a local minimum. The SD filter effectively controls the underlying image structure at different scales, and can handle a variety of types of data from different sensors. It is robust to outliers and other artifacts such as gradient reversal and global intensity shift, and has good edge-preserving smoothing properties. We demonstrate the flexibility and effectiveness of the proposed SD filter in a variety of applications, including depth upsampling, scale-space filtering, texture removal, flash/non-flash denoising, and RGB/NIR denoising.
BibTeX
@article{ham2018robust,
  author = {Bumsub Ham and Minsu Cho and Jean Ponce},
  title = {{Robust Guided Image Filtering Using Nonconvex Potentials}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2018},
  url = {https://openreview.net/forum?id=B8m9rDRAt1j},
}
Figure