Benchmarking Deep Inverse Models over time, and the Neural-Adjoint method

Simiao Ren, Willie Padilla, Jordan M. Malof

Deep learning models for inverse problems—finding the parameters that produce a specific measurement—are essentially stochastic search engines. This paper introduces a new time-dependent metric to evaluate these models and demonstrates that a simple gradient-based approach called the Neural-Adjoint (NA) method, when paired with a novel "boundary loss," consistently outperforms complex generative models like conditional Invertible Neural Networks (cINNs) and Variational Autoencoders (VAEs).

What Problem Does This Paper Address?

Inverse problems are ubiquitous in science and engineering, such as designing a meta-material to produce a specific light spectrum. They are notoriously difficult because they are often "ill-posed": multiple different sets of input parameters ($x$) can produce the exact same output ($y$).

While many deep learning models have been proposed to solve these challenges, the authors identify two major gaps in how we evaluate them:

  1. The "Single Guess" Fallacy: Most researchers evaluate a model based on a single predicted solution. In practice, engineers often generate many candidate solutions and use a simulator to pick the best one.
  2. Search Efficiency: We don't fully understand whether existing models fail because they can't find the right "neighborhood" of solutions, or because they can't precisely pinpoint the optimal values within those neighborhoods.

Methods: The Neural-Adjoint Approach

The authors conceptualize inverse modeling as a stochastic search. They propose a new metric, $r_T$, which measures the expected minimum error after drawing $T$ samples from a model.

To tackle the search effectively, they champion the Neural-Adjoint (NA) method. The process is straightforward:

  1. Train a Surrogate: Train a standard neural network to approximate the forward process ($y = f(x)$).
  2. Inverse via Backprop: To find $x$ for a target $y$, fix the network's weights and use gradient descent to "wiggle" the input $x$ until the predicted $y$ matches the target.
  3. The Boundary Loss ($L_{bnd}$): This is the paper's key technical contribution. Surrogate models are only accurate within the range of their training data. Without a constraint, gradient descent often pushes $x$ into "extrapolation zones" where the surrogate makes confident but false predictions. The boundary loss penalizes the model for searching outside the training data's bounding box.
The Neural-Adjoint method training and inference phases, highlighting the use of boundary loss.

Datasets

The authors evaluated their approach on four benchmarks:

  • D1: Ballistics: A 4D input task involving projectile trajectories.
  • D2: 2D Sinusoid (New): A simple 2D task used primarily for visualizing how different models explore the solution space.
  • D3: Robotic Arm: A 4D task involving joint angles for a 2D arm.
  • D4: Meta-material Design (New): A challenging, high-dimensional task (8D input, 300D output) involving the design of nanostructures to produce specific electromagnetic spectra. The authors provided a fast "proxy simulator" to make this accessible to other researchers.

Key Insights and Ideas

  • Performance is Time-Dependent: A model that looks best when you only have time for one guess (like the Tandem model) might be overtaken by others (like cINNs) if you have the computational budget to sample 50 times.
  • Generative Models vs. Gradient Descent: Models like cVAE and cINN are good at finding the general area of a solution but often suffer from "noise" or imprecision. The NA method, by contrast, uses the surrogate's gradients to "hone in" on the exact local minimum, leading to much higher accuracy.
  • The "Lying" Surrogate: The primary failure mode for the Neural-Adjoint method is its tendency to find "adversarial" solutions—points where the surrogate thinks the error is zero, but the real-world physics (or simulator) says otherwise. The Boundary Loss is a simple but highly effective "guardrail" against this.

Key Results

  • Superior Accuracy: The Neural-Adjoint (NA) method achieved the lowest (or nearly lowest) error across all four tasks, especially as the number of allowed inferences ($T$) increased.
  • Asymptotic Performance: In the meta-material task (D4), the NA method's error was orders of magnitude lower than its competitors (e.g., 6.67e-5 for NA vs. 4.45e-4 for the next best, cINN).
  • The Trade-off: The NA method is significantly slower than generative models. For 1,000 solutions on the meta-material task, NA took ~46 seconds, while the cVAE took less than half a second.
Performance comparison across four datasets showing the Neural-Adjoint (NA) method achieving the lowest error as the number of inferences increases.

Limitations and Future Work

  • Computational Cost: Because NA requires iterative gradient descent for every new target $y$, it may be too slow for real-time applications where sub-millisecond responses are required.
  • Boundary Geometry: The current boundary loss assumes the training data occupies a "hyper-cube." If the valid parameter space is highly non-convex or complex, a simple bounding box might not be sufficient.
  • Surrogate Dependency: The entire method hinges on the quality of the forward surrogate. If the surrogate cannot capture the underlying physics accurately, the adjoint gradients will lead the search astray.

The authors have released their code and simulators to encourage more robust benchmarking in the field: https://github.com/BensonRen/BDIMNNA

Summary updated

Abstract

We consider the task of solving generic inverse problems, where one wishes to determine the hidden parameters of a natural system that will give rise to a particular set of measurements. Recently many new approaches based upon deep learning have arisen generating impressive results. We conceptualize these models as different schemes for efficiently, but randomly, exploring the space of possible inverse solutions. As a result, the accuracy of each approach should be evaluated as a function of time rather than a single estimated solution, as is often done now. Using this metric, we compare several state-of-the-art inverse modeling approaches on four benchmark tasks: two existing tasks, one simple task for visualization and one new task from metamaterial design. Finally, inspired by our conception of the inverse problem, we explore a solution that uses a deep learning model to approximate the forward model, and then uses backpropagation to search for good inverse solutions. This approach, termed the neural-adjoint, achieves the best performance in many scenarios.
BibTeX
@inproceedings{ren2020benchmarking,
  author = {Simiao Ren and Willie Padilla and Jordan M. Malof},
  title = {{Benchmarking Deep Inverse Models over time, and the Neural-Adjoint method}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
}
Figure