Fixing Implicit Derivatives: Trust-Region Based Learning of Continuous Energy Functions

Chris Russell, Matteo Toso, Neill D. F. Campbell

Introduction

In computer vision and robotics, many tasks are framed as inverse problems: given observed data, the goal is to find hidden parameters (like 3D coordinates or camera poses) that best explain it. Traditionally, this is achieved by minimizing an "energy function"—a mathematical representation of how well a solution fits the data and physical priors.

While deep learning is incredibly powerful, classical energy-based methods are often more interpretable and grounded in physics. However, combining the two is notoriously difficult. The "layer of indirection" created by the optimization step makes it hard to perform backpropagation. Previous attempts to differentiate through these solvers using the implicit function theorem often suffered from numerical instability. If the energy landscape was too flat, the gradients would "explode," causing the training process to fail.

This paper introduces Wibergian Learning, a framework for training models that involve continuous energy minimization end-to-end. By characterizing the solution of an optimization problem as a fixed point of a trust-region method, the authors provide a stable way to compute gradients through the optimization process, effectively "fixing" the exploding gradient issues.

Methods: Wibergian Learning

The core innovation of Wibergian Learning lies in how it handles the "backward pass" (calculating gradients) through an optimization layer.

Wibergian Learning Framework

The approach relies on three key principles:

  1. Optimization as a Fixed Point: Instead of treating the minimum $y^*$ as an opaque black box, the method describes it as a fixed point of an optimization algorithm.
  2. Trust-Region Robustness: Standard Newton-step approaches use the inverse of the Hessian matrix ($H^{-1}$). When the Hessian is near zero (indicating a flat region), the gradient blows up. Wibergian Learning replaces this with a trust-region formulation: $(H + \lambda I)^{-1}$.
  3. Damped Gradients: The addition of the $\lambda I$ term (where $\lambda$ is a small constant) acts as a dampener. This bounds the magnitude of the gradient update, ensuring that even if the optimization landscape is problematic, the learning process remains stable.

This formulation allows a neural network to learn the weights and parameters of an energy function—such as the shape of a kernel or the importance of a specific prior—while the actual task is solved by a classical optimizer.

Key Insights and Results

The authors evaluated their approach on two distinct computer vision challenges, demonstrating significant improvements in stability and efficiency.

  • Stability over Pure Calculus: The fundamental insight is that the pure implicit function theorem is too fragile for deep learning. Grounding the derivative in a robust optimization algorithm (trust-region) provides the mathematical benefits without the numerical crashes. This damped gradient update prevents the "exploding gradient" problem, allowing for the use of standard stochastic gradient descent (SGD) with high momentum.
  • Tracker Fusion (VOT2018): The model was trained to optimally combine the outputs of 72 different tracking algorithms. Wibergian Learning achieved state-of-the-art results with an Intersection over Union (IoU) of 0.565, outperforming the best individual tracker (0.530). Crucially, the previous state-of-the-art implicit derivative method failed to converge on this task.
  • 3D Human Pose Estimation (Human3.6M): In multiview reconstruction, the method reduced error by 34mm compared to the baseline and 6mm over the previous best hand-tuned approach.
  • Efficiency vs. Grid Search: In the 3D pose estimation task, the gradient-based approach reached a solution in approximately 120 core hours. Achieving the same result via traditional grid search would have taken an astronomical $5 \cdot 10^{6648}$ hours.

Ultimately, this hybrid interpretability allows researchers to keep their geometric models (like Structure from Motion) but use deep learning to "tune" them automatically, rather than relying on hand-crafted heuristics.

Limitations and Future Work

While highly effective, Wibergian Learning has a few limitations:

  • Local Minima: Like all gradient-based methods, it can still get stuck in local minima if the energy function is poorly initialized.
  • Vanishing Gradients: If the energy landscape is too narrow, the "stuck neuron" problem can occur, where the parameters no longer influence the location of the minimum.

For future work, the authors suggest this framework could be applied to Capsule Networks (treating routing as an energy problem) or used to improve the 3D accuracy of Structure from Motion (SfM) pipelines by integrating them directly into end-to-end training.

Summary updated

Abstract

We present a new technique for the learning of continuous energy functions that we refer to as Wibergian Learning. One common approach to inverse problems is to cast them as an energy minimisation problem, where the minimum cost solution found is used as an estimator of hidden parameters. Our new approach formally characterises the dependency between weights that control the shape of the energy function, and the location of minima, by describing minima as fixed points of optimisation methods. This allows for the use of gradient-based end-to- end training to integrate deep-learning and the classical inverse problem methods. We show how our approach can be applied to obtain state-of-the-art results in the diverse applications of tracker fusion and multiview 3D reconstruction.
BibTeX
@inproceedings{russell2019fixing,
  author = {Chris Russell and Matteo Toso and Neill D. F. Campbell},
  title = {{Fixing Implicit Derivatives: Trust-Region Based Learning of Continuous Energy Functions}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
}
Figure