Benchmarking deep inverse models over time, and the neural-adjoint method

Simiao Ren, Willie Padilla, Jordan Malof

TL;DR

This paper reframes deep inverse modeling as a stochastic search process, arguing that model performance should be evaluated as a function of time (number of samples) rather than a single estimate. The authors introduce the Neural-Adjoint (NA) method—which uses a surrogate forward model and a novel "boundary loss"—demonstrating that it consistently outperforms state-of-the-art generative models across diverse benchmarks, including a complex new meta-material design task.

What Problem Does This Paper Address?

Inverse problems—determining the hidden parameters ($x$) that produce a specific measurement ($y$)—are ubiquitous in science and engineering, from medical imaging to material design. These problems are often "ill-posed" because they are one-to-many: multiple different parameter sets can produce the same result.

While deep learning models like Variational Auto-Encoders (VAEs) and Invertible Neural Networks (INNs) have shown promise, the authors identify two major gaps:

  1. Metric Mismatch: Most researchers evaluate models based on a single prediction. However, since these models are often stochastic, they should be judged on their ability to find a good solution given a certain amount of "search time" (multiple samples).
  2. Search Inefficiency: Existing generative models often fail to either fully explore the parameter space or precisely localize the optimal solution, leading to residual errors.

Methods

The authors propose the Neural-Adjoint (NA) method, which breaks the problem into two distinct phases:

  1. Surrogate Training: A standard neural network is trained to approximate the forward process ($y = f(x)$). This creates a fast, differentiable "proxy" of the real-world system.
  2. Inverse Search via Backprop: To find $x$ for a target $y$, the model fixes the network weights and uses backpropagation to adjust the input $x$ iteratively. By starting from multiple random initializations, the model searches the parameter space for the best fit.
Figure 1: Architecture of Neural Adjoint method

A critical innovation is the Boundary Loss. Surrogate models are notoriously inaccurate outside their training data range. The boundary loss penalizes the search if it wanders into these "untrusted" regions, ensuring the optimizer stays within the domain where the surrogate is reliable.

Datasets

The study benchmarks five models across four tasks:

  • D1: Ballistics: A 4D input task predicting the landing position of a projectile.
  • D2: 2D Sinusoid: A simple task used primarily for visualizing how different models explore the parameter space.
  • D3: Robotic Arm: A 4D task involving the geometry of a three-joint arm.
  • D4: Meta-material Design (New): A high-dimensional task (8D input, 300D output) where the goal is to design cylinder geometries that produce a specific electromagnetic reflection spectrum.
Figure 2: Illustration of the meta-material problem

Key Insights and Ideas

  • Performance is Time-Dependent: The "best" model changes depending on how many samples ($T$) you are allowed to draw. For instance, a model that is accurate with one sample might be overtaken by another that explores the space better as $T$ increases.
  • Exploration vs. Localization: Visualization on the 2D sinusoid task revealed that while models like cINNs and VAEs find the general region of solutions, they often produce "noisy" results. In contrast, the NA method precisely localizes the optimal rings of solutions.
  • The Power of Boundary Constraints: Without the boundary loss, the NA method often converges to "adversarial" solutions—points outside the training data where the surrogate thinks the error is zero, but the true simulator shows high error.

Key Results

The Neural-Adjoint method achieved the lowest error in nearly all scenarios, particularly excelling in the high-dimensional meta-material task.

Figure 3: (a-d) Performance on each model for each benchmark task as a function of T.
  • Accuracy: In the meta-material task (D4), NA's error was nearly an order of magnitude lower than the next best model (cINN) across all values of $T$.
  • Efficiency vs. Speed: The primary trade-off is inference time. Because NA requires iterative gradient descent, it is significantly slower than generative models. For 1,000 solutions in the meta-material task, NA took ~46 seconds, while other models took less than 1 second.
Table 2: Total Inference time (t) in seconds for 1,000 solutions

Limitations and Future Work

  • Computational Cost: The iterative nature of NA makes it less suitable for real-time applications compared to "one-shot" generative models.
  • Domain Assumptions: The current boundary loss assumes the training data occupies a simple hyper-cube. If the valid parameter space is highly non-convex or complex, this loss might be less effective.
  • Surrogate Uncertainty: The authors suggest that incorporating the uncertainty of the surrogate model (e.g., via Bayesian neural networks) into the search process could further improve reliability.

Summary updated

Abstract

We consider the task of solving generic inverse problems, where one wishes to determine the hidden parameters of a natural system that will give rise to a particular set of measurements. Recently many new approaches based upon deep learning have arisen generating impressive results. We conceptualize these models as different schemes for efficiently, but randomly, exploring the space of possible inverse solutions. As a result, the accuracy of each approach should be evaluated as a function of time rather than a single estimated solution, as is often done now. Using this metric, we compare several state-of-the-art inverse modeling approaches on four benchmark tasks: two existing tasks, one simple task for visualization and one new task from metamaterial design. Finally, inspired by our conception of the inverse problem, we explore a solution that uses a deep learning model to approximate the forward model, and then uses backpropagation to search for good inverse solutions. This approach, termed the neural-adjoint, achieves the best performance in many scenarios.
BibTeX
@inproceedings{ren2020benchmarking,
  author = {Simiao Ren and Willie Padilla and Jordan Malof},
  title = {{Benchmarking deep inverse models over time, and the neural-adjoint method}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
}
Figure