Abstract
Motivated by indirect measurements and applications from nanometrology with a mixed noise model, we develop a novel algorithm for jointly estimating the posterior and the noise parameters in Bayesian inverse problems. We propose to solve the problem by an expectation maximization (EM) algorithm. Based on the current noise parameters, we learn in the E-step a conditional normalizing flow that approximates the posterior. In the M-step, we propose to find the noise parameter updates again by an EM algorithm, which has analytical formulas. We compare the training of the conditional normalizing flow with the forward and reverse KL, and show that our model is able to incorporate information from many measurements, unlike previous approaches.
Introduction
In fields such as nanometrology, researchers are tasked with characterizing structures at the atomic scale using indirect measurements. This process, known as a Bayesian inverse problem, involves determining the true physical parameters ($X$) of a system from observed data ($Y$) through a forward physical model ($F$). However, two primary challenges complicate this task: complex noise profiles and the inherent multimodality of the parameter space.
Traditional models often assume simple additive noise, but real-world instruments frequently exhibit "mixed noise," where the error scales with the signal itself (multiplicative noise). Furthermore, physical systems are often non-unique; different parameter sets can produce nearly identical measurements, leading to multimodal posterior distributions. We introduce Conditional DeepGEM, a nested Expectation-Maximization (EM) framework designed to jointly estimate both the system parameters and the noise characteristics while remaining robust to multimodality.
Methodology
The Mixed Noise Model
We consider a measurement model where the observation $Y$ is affected by both additive and multiplicative components: $$Y = F(X) + (a F(X) + b) \odot \epsilon$$ where $a$ and $b$ represent the noise parameters and $\epsilon$ is standard Gaussian noise. Estimating $a$ and $b$ is critical for accurate posterior estimation, yet they are typically unknown a priori.
Nested EM Framework
Conditional DeepGEM utilizes a nested EM structure to iteratively refine its estimates:
- The E-Step (Posterior Estimation): We employ a Conditional Normalizing Flow (CNF) to approximate the posterior distribution $p(X|Y)$. By making the flow conditional, the model is "amortized," meaning once it is trained, it can provide a posterior for any new measurement $y$ without retraining. Crucially, we utilize Forward KL divergence for training. While standard generative models often use Reverse KL (which is "mode-seeking" and tends to collapse onto a single peak), Forward KL is "mode-covering," ensuring that all physically valid solutions in a multimodal landscape are captured.
- The M-Step (Noise Parameter Update): To update the noise parameters $a$ and $b$, we derived a specific "inner" EM algorithm. This approach provides analytical, closed-form update rules. This is significantly more stable and computationally efficient than using stochastic gradient descent for hyperparameter tuning, which can be sensitive to initialization and learning rates.
Experimental Evaluation
The algorithm was evaluated on two challenging benchmarks in nano-optics:
- EUV Photomask: A 3-parameter problem involving extreme ultraviolet light refraction. This benchmark is characterized by a highly multimodal posterior, making it a rigorous test for mode-covering capabilities.
- Line Grating with Oxide Layer: A more complex 7-parameter problem. To handle the computationally expensive Finite Element Method (FEM) simulations required for the forward operator, we utilized surrogate neural networks trained on data from JCMsuite.
Key Results and Insights
Amortization and Information Gain
By utilizing a conditional framework, the model can incorporate information from multiple measurements simultaneously. In the EUV Photomask experiments, the error in noise parameter estimation decreased from 0.50 (with a single measurement) to 0.20 (with eight measurements). This demonstrates that the model effectively "learns" the noise characteristics of the instrument as it processes more data.
Superiority of Forward KL
The choice of Forward KL divergence proved vital for physical accuracy. In multimodal scenarios, the Forward KL version of Conditional DeepGEM successfully mapped the entire landscape of possible solutions, whereas Reverse KL approaches often missed secondary modes. This was reflected in the Evidence Lower Bound (ELBO), which remained consistently higher for the Forward KL approach.
Stability via Analytical Updates
The analytical formulas derived for the M-step noise updates provided reliable convergence. By avoiding black-box optimization for the noise parameters, the algorithm ensures that the noise estimation remains mathematically grounded and stable across different datasets.
Conclusion and Future Work
Conditional DeepGEM represents a significant advancement in solving Bayesian inverse problems with complex noise. By combining the mode-covering properties of Forward KL with the efficiency of amortized Conditional Normalizing Flows and analytical EM updates, the framework provides a robust tool for high-precision metrology.
Future research will focus on transitioning from synthetic surrogate models to real-world experimental data. Additionally, while the current inner EM derivation assumes Gaussian-based mixed noise, extending this framework to non-Gaussian distributions remains a promising avenue for further development.
Summary updated