Dynamic sampling schemes for optimal noise learning under multiple\n nonsmooth constraints

Luca Calatroni, Juan Carlos De los Reyes, Carola‐Bibiane Schönlieb

Abstract: We consider the bilevel optimisation approach proposed by De Los Reyes, Schönlieb (2013) for learning the optimal parameters in a Total Variation (TV) denoising model featuring for multiple noise distributions. In applications, the use of databases (dictionaries) allows an accurate estimation of the parameters, but reflects in high computational costs due to the size of the databases and to the nonsmooth nature of the PDE constraints. To overcome this computational barrier we propose an optimisation algorithm that by sampling dynamically from the set of constraints and using a quasi-Newton method, solves the problem accurately and in an efficient way.

1. Introduction

Total Variation (TV) denoising is a fundamental technique in image processing, valued for its ability to remove noise while preserving critical edge information. However, the performance of TV models is highly sensitive to the regularization parameter $\lambda$. Selecting an optimal $\lambda$ often involves bilevel optimization, where the outer loop searches for the parameter and the inner loop solves a denoising problem across a large database of images. When dealing with hundreds or thousands of images, the requirement to solve a nonsmooth partial differential equation (PDE) for every image at every optimization step creates a significant "computational barrier."

2. Methodology: Dynamic Sampling BFGS

To address this inefficiency, we propose a dynamic sampling strategy integrated with a quasi-Newton (BFGS) optimization method. The core innovation lies in adaptively selecting the number of training samples used in each iteration.

2.1 Dynamic Sample Size and Variance Condition

Rather than processing the entire database $N$ from the start, the algorithm begins with a small subset $S_0$. The sample size is increased only when the variance of the stochastic gradient is too high to ensure a descent direction. By monitoring this variance relative to the gradient magnitude, the algorithm maintains computational efficiency in early iterations where high precision is unnecessary.

2.2 Quasi-Newton Optimization

The BFGS method is employed to approximate second-order Hessian information, allowing for larger and more accurate steps compared to standard gradient descent. This approach is specifically adapted to handle the nonlinear and nonsmooth constraints inherent in the TV denoising solution map.

3. Experimental Setup

The proposed method was evaluated using both simulated datasets and real-world medical data. Specifically, slices from the OASIS MRI database were used to test the algorithm's performance on complex anatomical structures.

The model was tested against various noise profiles, including standard Gaussian noise and mixed noise (Gaussian plus impulse/salt-and-pepper noise).

4. Results and Discussion

The dynamic sampling approach demonstrates substantial computational savings without sacrificing the quality of the learned parameters.

As shown in the table above, for a database of $N=100$ images, the standard approach required 1,400 effective PDE solves. In contrast, the dynamic sampling method achieved a comparable result with only 520 solves—a reduction of approximately 63%. Furthermore, the difference between the estimated $\lambda$ and the true optimal value remained consistently low, typically under 5%.

5. Conclusion

This work successfully adapts dynamic sampling techniques to the challenging domain of nonlinear, nonsmooth bilevel optimization. By prioritizing speed in early iterations and increasing precision only as needed, the algorithm overcomes the computational barriers associated with large-scale image databases. Future research will focus on establishing a more rigorous convergence theory for this framework and exploring sparsity-promoting regularization for multi-parameter learning.

Summary updated

Abstract

We consider the bilevel optimisation approach proposed by De Los Reyes,\nSch\\"onlieb (2013) for learning the optimal parameters in a Total Variation\n(TV) denoising model featuring for multiple noise distributions. In\napplications, the use of databases (dictionaries) allows an accurate estimation\nof the parameters, but reflects in high computational costs due to the size of\nthe databases and to the nonsmooth nature of the PDE constraints. To overcome\nthis computational barrier we propose an optimisation algorithm that by\nsampling dynamically from the set of constraints and using a quasi-Newton\nmethod, solves the problem accurately and in an efficient way.\n
BibTeX
@inproceedings{calatroni2013dynamic,
  author = {Luca Calatroni and Juan Carlos De los Reyes and Carola‐Bibiane Schönlieb},
  title = {{Dynamic sampling schemes for optimal noise learning under multiple\textbackslash\{\}n nonsmooth constraints}},
  booktitle = {System Modelling and Optimization},
  year = {2013},
  doi = {10.1007/978-3-662-45504-3\_8},
}

Loading related content...

Figure