TL;DR
This paper introduces a nested Expectation-Maximization (EM) algorithm that jointly estimates both the hidden parameters of a physical system and the complex "mixed" noise (additive + multiplicative) affecting measurements. By leveraging Conditional Normalizing Flows and a shift to Forward KL divergence, the authors create a "mode-covering" solver that can incorporate multiple measurements simultaneously without needing to retrain for every new observation.
What Problem Does This Paper Address?
In fields like nanometrology (measuring structures at the atomic scale), researchers face "Bayesian inverse problems." You have a measurement ($Y$), a physical model ($F$), and you want to find the true parameters ($X$) that created that measurement.
However, two major hurdles usually stand in the way:
- Complex Noise: Real-world instruments don't just have simple background hiss (additive noise). They often have "mixed noise," where the error scales with the signal itself (multiplicative noise).
- Multimodality: In many physical systems, different sets of parameters can produce nearly identical measurements. Standard machine learning tools often suffer from "mode collapse," where the algorithm "picks a favorite" solution and ignores other equally valid possibilities.
Previous approaches like the original DeepGEM were "one-off" solutions—they had to be retrained for every single new measurement and struggled with multimodal distributions.
Methods
The authors propose an extension called Conditional DeepGEM, which uses a nested EM framework:
- The E-Step (Posterior Estimation): Instead of a standard Normalizing Flow, they use a Conditional Normalizing Flow (CNF). This allows the model to be "amortized"—once trained, it can provide a posterior distribution for any new measurement $y$ instantly. Crucially, they train this using Forward KL divergence. While Reverse KL is common in generative AI, it is "mode-seeking" (it narrows in on one peak). Forward KL is "mode-covering," ensuring the model captures all possible physical solutions.
- The M-Step (Noise Parameter Update): To estimate the noise levels (parameters $a$ and $b$), the authors derived a specific "inner" EM algorithm. Unlike standard gradient descent, which can be finicky, this inner EM provides analytical (closed-form) update rules, making the estimation of noise parameters both fast and mathematically stable.
Datasets
The researchers evaluated their method on two challenging problems from nano-optics:
- EUV Photomask: A 3-parameter problem involving extreme ultraviolet light refraction. This is a classic benchmark known for having a multimodal posterior (multiple "correct" answers).
- Line Grating with Oxide Layer: A more complex 7-parameter problem. Because the physical simulations (Finite Element Method) are computationally expensive, the authors used surrogate neural networks trained on data from the JCMsuite software to act as the forward operator.
Key Insights and Ideas
- Amortization Matters: By making the model conditional, the authors show that the system can "learn" from many measurements at once. This allows the model to become more certain about the noise parameters as it sees more data.
- The Forward KL Advantage: The paper provides a clear visual demonstration (Figure 2) of why Forward KL is superior for physics. While the Reverse KL version often misses secondary modes, the Forward KL version successfully maps out the entire landscape of possible solutions.
- Analytical Stability: By deriving exact formulas for the noise updates in the M-step, the authors avoid the "black-box" nature of stochastic gradient descent for hyperparameter tuning, leading to more reliable convergence.
Key Results
- Better Noise Estimation: In the EUV Photomask test, the Forward Conditional DeepGEM reduced the error in noise parameter estimation significantly as the number of measurements increased (from an error of 0.50 with one measurement down to 0.20 with eight).
- Higher ELBO: The Evidence Lower Bound (ELBO)—a measure of how well the model fits the data—was consistently higher for the Forward KL approach compared to Reverse KL and traditional grid-search methods.
- Robustness to Multimodality: The model successfully identified multiple "modes" (peaks of probability) in the posterior, which is vital for researchers who need to know if their measurement has multiple valid interpretations.
Limitations and Future Work
- Synthetic vs. Real Data: The study currently relies on synthetic measurements and surrogate models. The authors acknowledge that the next step is testing the algorithm on "noisy" real-world experimental data.
- Noise Distribution Assumptions: The current "inner EM" derivation assumes Gaussian-based mixed noise. While this covers many use cases, extending the math to non-Gaussian noise distributions remains a task for the future.
- Initialization Sensitivity: While the "logdet schedule" helps prevent the model from missing modes early in training, the authors note it doesn't solve the problem perfectly in every single scenario.
Summary updated