Parametric Majorization for Data-Driven Energy Minimization Methods

Jonas Geiping, Michael MoellerICCV 2019 · OpenReview

Abstract

Energy minimization methods are a classical tool in a multitude of computer vision applications. While they are interpretable and well-studied, their regularity assumptions are difficult to design by hand. Deep learning techniques on the other hand are purely data-driven, often provide excellent results, but are very difficult to constrain to predefined physical or safety-critical models. A possible combination between the two approaches is to design a parametric energy and train the free parameters in such a way that minimizers of the energy correspond to desired solution on a set of training examples. Unfortunately, such formulations typically lead to bi-level optimization problems, on which common optimization algorithms are difficult to scale to modern requirements in data processing and efficiency. In this work, we present a new strategy to optimize these bi-level problems. We investigate surrogate single-level problems that majorize the target problems and can be implemented with existing tools, leading to efficient algorithms without collapse of the energy function. This framework of strategies enables new avenues to the training of parameterized energy minimization models from large data.

Introduction: The Bi-level Optimization Challenge

Training computer vision models often feels like a choice between two worlds. On one hand, variational methods (energy minimization) are interpretable, physically grounded, and mathematically robust, but designing their parameters by hand is notoriously difficult. On the other hand, deep learning is incredibly powerful and data-driven but acts as a "black box" that is hard to constrain to physical laws.

The middle ground—learning the parameters of an energy-based model from data—usually leads to a complex mathematical challenge called bi-level optimization. The core problem is to find parameters $\theta$ such that the image $x^*$ that minimizes the energy $E(x, \theta)$ is as close as possible to the ground truth: $$\min_{\theta} \text{Loss}(\arg\min_x E(x, \theta), \text{GroundTruth})$$

The nested "$\arg\min$" is the primary bottleneck. Standard gradient descent requires differentiating through the entire minimization process, which is computationally expensive, memory-intensive, and often numerically unstable for the non-smooth energy functions common in vision tasks.

Methodology: Parametric Majorization

This paper introduces Parametric Majorization to bypass the bi-level bottleneck. Instead of solving the bi-level problem directly, we identify a surrogate function $S(\theta)$ that acts as an upper bound (a majorizer) to the original loss. Minimizing this surrogate guarantees progress on the original loss while being significantly easier to compute.

Visualization of various surrogate functions (Bregman, Partial) majorizing the original loss function.

Key Innovations:

  • Bregman Surrogates: We leverage Bregman distances—a generalized distance measure based on convex functions. We demonstrate that if the loss function is bounded by the Bregman distance of the energy function, the optimization objective simplifies into a single-level problem.
  • Dual Reformulation: Using Fenchel duality, we transform the complex saddle-point problem into a pure minimization problem that standard solvers can handle efficiently.
  • Gradient Penalty: We derive a version where penalizing the gradient of the energy function at the ground truth prevents "energy collapse," where a model might otherwise learn a trivial zero-energy solution.
  • Iterative Majorization: We introduce an iterative scheme that repeatedly tightens the surrogate bound, ensuring stable and monotonic descent of the true bi-level loss.

Experimental Results

We validated our approach across three distinct tasks: Computed Tomography (CT) using Shepp-Logan phantoms, Variational Segmentation on the Cityscapes dataset, and Image Denoising using the BSDS dataset to learn optimal convolutional filters.

Performance and Efficiency

The proposed method achieved an order-of-magnitude speedup compared to state-of-the-art implicit differentiation methods. For example, in the denoising task, training 96 filters (9x9) was completed in approximately 2 hours, whereas previous methods reportedly required up to 20 days for similar configurations.

Beyond speed, the majorization framework delivered tangible performance gains:

  • CT Reconstruction: The learned correction term improved the Peak Signal-to-Noise Ratio (PSNR) by nearly 2dB over standard hand-tuned methods.
  • Segmentation: The iterative majorization scheme significantly increased accuracy compared to baseline models that did not account for the perimeter (Total Variation) constraints during training.
  • Scalability: Unlike unrolling methods, the memory cost of this approach does not scale with the complexity of the inference algorithm, making it suitable for large-scale data.

Discussion and Limitations

While Parametric Majorization offers a faster way to train interpretable models, it currently relies on the lower-level energy function being convex in $x$. Extending this framework to non-convex energies remains a major open challenge. Additionally, while the iterative scheme helps tighten the surrogate bounds, the choice of surrogate can still impact the final parameter quality. Future work will explore integrating more complex architectures, such as Transformers, within this energy-minimization framework.

Summary updated

Abstract

Energy minimization methods are a classical tool in a multitude of computer vision applications. While they are interpretable and well-studied, their regularity assumptions are difficult to design by hand. Deep learning techniques on the other hand are purely data-driven, often provide excellent results, but are very difficult to constrain to predefined physical or safety-critical models. A possible combination between the two approaches is to design a parametric energy and train the free parameters in such a way that minimizers of the energy correspond to desired solution on a set of training examples. Unfortunately, such formulations typically lead to bi-level optimization problems, on which common optimization algorithms are difficult to scale to modern requirements in data processing and efficiency. In this work, we present a new strategy to optimize these bi-level problems. We investigate surrogate single-level problems that majorize the target problems and can be implemented with existing tools, leading to efficient algorithms without collapse of the energy function. This framework of strategies enables new avenues to the training of parameterized energy minimization models from large data.
BibTeX
@inproceedings{geiping2019parametric,
  author = {Jonas Geiping and Michael Moeller},
  title = {{Parametric Majorization for Data-Driven Energy Minimization Methods}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=7iq9-fs6ewN},
}
Figure