Abstract
This paper introduces Probabilistic Surrogate Networks (PSNs), a framework for creating fast, neural-network-based approximations of complex stochastic simulators. Unlike traditional surrogates that treat simulators as black-box functions, PSNs mimic the internal control flow and logic of the original program. This allows them to handle "unbounded randomness"—scenarios where the number of random decisions made by a simulator can change or grow infinitely with every run. PSNs achieve massive speedups (up to 90x) while maintaining high fidelity to the original simulator's logic and posterior distributions.
Introduction
Scientific and engineering simulators, ranging from weather forecasting to aerospace manufacturing, are often stochastic. To perform tasks like sensitivity analysis or Bayesian inference, researchers must run these computationally expensive simulators thousands of times. Traditional surrogate modeling attempts to replace these simulators with simpler machine learning models. However, most existing methods fail when the simulator is a complex program where the number of internal random variables is not fixed, such as in simulators with loops or branches that depend on random outcomes. This challenge is known as unbounded randomness.
Methodology: Mirroring Program Logic
The core innovation of PSNs is that the surrogate is structured to mirror the execution "trace" of the original probabilistic program.
RNN Core and Dynamic Expansion
PSNs utilize Recurrent Neural Networks (RNNs) to maintain a "memory" of the simulator's state. This allows the model to predict the next random variable or control-flow branch based on the sequence of preceding events. To handle the dynamic nature of program execution, the authors developed a mathematical framework for Dynamic Expansion.
As shown in Figure 1, the network can "grow" its output layers to accommodate new execution paths (addresses) discovered during training. This expansion is measure-preserving, ensuring the model learns new paths without distorting the probabilities of previously learned ones. The framework is integrated with PyProb, a universal probabilistic programming library, allowing it to intercept and learn from any simulator written in standard languages like Python or C++.
Experiments and Results
The researchers evaluated PSNs on three distinct types of simulators: synthetic stochastic control flow, program synthesis (generating Python code), and the RAVEN simulator used in aerospace manufacturing for composite material curing.
High Fidelity and Speed
In the program synthesis task, the PSN generated valid Python programs 99.62% of the time, demonstrating its ability to learn complex underlying grammars. In the RAVEN experiment, the PSN achieved a 90.16x speedup for simulation and a 15.32x speedup for posterior inference compared to the original simulator.
As illustrated in Figure 4, the posterior distributions (the "answers" provided by the model) were nearly indistinguishable from the ground truth. The heatmaps in Figure 4(c) further confirm that the PSN successfully learned the transition probabilities between different program addresses, effectively capturing the simulator's internal logic.
Key Insights
- Structural Compatibility: By following the same "addressing" logic as the simulator, the surrogate remains interpretable and compatible with existing inference tools.
- Amortized Inference: The speed of the PSN allows for "Bayesian inversion" (guessing hidden causes from observed results) to be performed an order of magnitude faster than traditional methods.
- Universal Approximation: PSNs break the "fixed-dimension" barrier of standard neural networks, treating simulator execution as a sequence of events of arbitrary length.
Limitations and Future Work
Despite their success, PSNs have limitations. They may assign zero probability to extremely rare execution paths not encountered during training (under-approximation). Additionally, while RNNs are powerful, simulators requiring truly infinite memory (like an unbounded stack) might require more advanced architectures like Transformers or Differentiable Neural Computers. Finally, the computational complexity scales with the number of unique execution paths, which could be a bottleneck for extremely complex branching programs.
Summary updated