Learning Instance-Specific Parameters of Black-Box Models Using Differentiable Surrogates

Arnisha Khondaker, Nilanjan Ray

Introduction

Many powerful engineering tools and algorithms are "black boxes": their internal logic is non-differentiable or proprietary, preventing the direct calculation of gradients. Tuning the parameters of these models—such as an Image Signal Processor (ISP) or a complex simulator—traditionally requires tedious manual effort or exhaustive grid searches.

Crucially, most existing methods find a single "best" set of parameters for an entire dataset. However, in tasks like image denoising, a dark, grainy photo requires different settings than a bright, blurry one. This paper addresses the challenge of instance-specific parameter tuning, finding optimal settings for every individual input automatically.

Methodology

The authors propose a three-step approach to bridge the gap between deep learning and non-differentiable algorithms:

  1. The Surrogate Model: A UNETR (a hybrid architecture combining Transformers and U-Nets) acts as a differentiable surrogate, trained to mimic the behavior of the black-box (e.g., the BM3D denoiser). Because the surrogate is a neural network, gradients can be calculated through it.
  2. The Parameter Learner: A second network (a standard U-Net) takes a noisy image as input and predicts the optimal parameters for the black-box.
  3. End-to-End Training (Algorithm 3): The system is trained in a loop where the Parameter Learner predicts settings, and the Surrogate estimates the output. Gradients flow back through the Surrogate to update the Parameter Learner. Simultaneously, the Surrogate is updated dynamically to stay "in sync" with the actual black-box's behavior as the search space evolves.
Overall framework for instance-specific parameter learning using a differentiable surrogate model.

To handle discrete settings (categorical data) within the gradient-based framework, the model is trained with continuous values and quantizes them only when calling the actual black-box, allowing for smooth exploration of the parameter space.

Experimental Results

The method was evaluated on the Smartphone Image Denoising Dataset (SIDD) and the Color Berkeley Segmentation Dataset (CBSD68). The instance-specific approach significantly outperformed standard proxy-based optimization and competitive random grid searches.

On the SIDD dataset, the proposed method achieved a PSNR of 36.51 dB, compared to 33.23 dB for standard proxy optimization and 36.00 dB for a heavy random grid search. The Structural Similarity Index (SSIM) reached 0.93, a notable improvement from the 0.87 baseline.

Qualitative comparison of denoising results: (a) Noisy input, (b) Global tuning, (c) Proposed instance-specific tuning.

Qualitative results demonstrate that instance-specific tuning preserves more texture and produces fewer artifacts than global tuning methods by allowing for localized adjustments of patch sizes and noise coefficients.

Conclusion

This framework demonstrates that "one size does not fit all" in algorithm tuning. By using differentiable surrogates that adapt during training, researchers can automatically optimize black-box models for individual inputs. Future work will focus on scaling this approach to higher-dimensional parameter spaces, such as full camera ISPs, and extending the framework to domains like Natural Language Processing and physical system modeling.

Summary updated

Abstract

Tuning parameters of a non-differentiable or black-box compute is challenging. Existing methods rely mostly on random sampling or grid sampling from the parameter space. Further, with all the current methods, it is not possible to supply any input specific parameters to the black-box. To the best of our knowledge, for the first time, we are able to learn input-specific parameters for a black box in this work. As a test application, we choose a popular image denoising method BM3D as our black-box compute. Then, we use a differentiable surrogate model (a neural network) to approximate the black-box behaviour. Next, another neural network is used in an end-to-end fashion to learn input instance-specific parameters for the black-box. Motivated by prior advances in surrogate-based optimization, we applied our method to the Smartphone Image Denoising Dataset (SIDD) and the Color Berkeley Segmentation Dataset (CBSD68) for image denoising. The results are compelling, demonstrating a significant increase in PSNR and a notable improvement in SSIM nearing 0.93. Experimental results underscore the effectiveness of our approach in achieving substantial improvements in both model performance and optimization efficiency. For code and implementation details, please refer to our GitHub repository: https://github.com/arnisha-k/instance-specific-param
BibTeX
@inproceedings{khondaker2025learning,
  author = {Arnisha Khondaker and Nilanjan Ray},
  title = {{Learning Instance-Specific Parameters of Black-Box Models Using Differentiable Surrogates}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00722},
}
Figure