Deep Probabilistic Surrogate Networks for Universal Simulator Approximation

Andreas Munk, Adam Scibior, Atilim Günes Baydin, Andrew Stewart, Goran Fernlund, Anoush Poursartip, Frank WoodarXiv 2019 · OpenReview

Abstract

This paper introduces Probabilistic Surrogate Networks (PSNs), a framework for creating fast, neural-network-based approximations of complex stochastic simulators. Unlike traditional surrogates that treat simulators as black-box functions, PSNs mimic the internal control flow and logic of the original program. This allows them to handle "unbounded randomness"—scenarios where the number of random decisions made by a simulator can change or grow infinitely with every run. PSNs achieve massive speedups (up to 90x) while maintaining high fidelity to the original simulator's logic and posterior distributions.

Introduction

Scientific and engineering simulators, ranging from weather forecasting to aerospace manufacturing, are often stochastic. To perform tasks like sensitivity analysis or Bayesian inference, researchers must run these computationally expensive simulators thousands of times. Traditional surrogate modeling attempts to replace these simulators with simpler machine learning models. However, most existing methods fail when the simulator is a complex program where the number of internal random variables is not fixed, such as in simulators with loops or branches that depend on random outcomes. This challenge is known as unbounded randomness.

Methodology: Mirroring Program Logic

The core innovation of PSNs is that the surrogate is structured to mirror the execution "trace" of the original probabilistic program.

RNN Core and Dynamic Expansion

PSNs utilize Recurrent Neural Networks (RNNs) to maintain a "memory" of the simulator's state. This allows the model to predict the next random variable or control-flow branch based on the sequence of preceding events. To handle the dynamic nature of program execution, the authors developed a mathematical framework for Dynamic Expansion.

Figure 1: The dynamic expansion mechanism allows the neural network to grow as it encounters new execution paths (addresses) during training, preserving existing probability measures.

As shown in Figure 1, the network can "grow" its output layers to accommodate new execution paths (addresses) discovered during training. This expansion is measure-preserving, ensuring the model learns new paths without distorting the probabilities of previously learned ones. The framework is integrated with PyProb, a universal probabilistic programming library, allowing it to intercept and learn from any simulator written in standard languages like Python or C++.

Experiments and Results

The researchers evaluated PSNs on three distinct types of simulators: synthetic stochastic control flow, program synthesis (generating Python code), and the RAVEN simulator used in aerospace manufacturing for composite material curing.

High Fidelity and Speed

In the program synthesis task, the PSN generated valid Python programs 99.62% of the time, demonstrating its ability to learn complex underlying grammars. In the RAVEN experiment, the PSN achieved a 90.16x speedup for simulation and a 15.32x speedup for posterior inference compared to the original simulator.

Figure 4: Evaluation on a stochastic control flow program (a) showing that the PSN accurately recovers the ground truth (GT) posterior distribution (b) and learns the correct address transition probabilities (c).

As illustrated in Figure 4, the posterior distributions (the "answers" provided by the model) were nearly indistinguishable from the ground truth. The heatmaps in Figure 4(c) further confirm that the PSN successfully learned the transition probabilities between different program addresses, effectively capturing the simulator's internal logic.

Key Insights

  • Structural Compatibility: By following the same "addressing" logic as the simulator, the surrogate remains interpretable and compatible with existing inference tools.
  • Amortized Inference: The speed of the PSN allows for "Bayesian inversion" (guessing hidden causes from observed results) to be performed an order of magnitude faster than traditional methods.
  • Universal Approximation: PSNs break the "fixed-dimension" barrier of standard neural networks, treating simulator execution as a sequence of events of arbitrary length.

Limitations and Future Work

Despite their success, PSNs have limitations. They may assign zero probability to extremely rare execution paths not encountered during training (under-approximation). Additionally, while RNNs are powerful, simulators requiring truly infinite memory (like an unbounded stack) might require more advanced architectures like Transformers or Differentiable Neural Computers. Finally, the computational complexity scales with the number of unique execution paths, which could be a bottleneck for extremely complex branching programs.

Summary updated

BibTeX
@misc{munk2019deep,
  author = {Andreas Munk and Adam Scibior and Atilim Günes Baydin and Andrew Stewart and Goran Fernlund and Anoush Poursartip and Frank Wood},
  title = {{Deep Probabilistic Surrogate Networks for Universal Simulator Approximation}},
  year = {2019},
  url = {https://openreview.net/forum?id=NIPrwlTNywV},
}
Figure