Neural Processes (NPs) represent a powerful class of models that combine the flexibility of deep learning with the principled uncertainty quantification of stochastic processes. Despite their theoretical appeal, NPs often suffer from underfitting and suboptimal performance in practical applications. This research identifies that these issues stem from an "ill-posed" training objective and proposes a novel framework: the Self-normalized Importance weighted Neural Process (SI-NP).
The Inference Gap Problem
Standard NP training relies on a "consistent regularizer" which, unlike the Evidence Lower Bound (ELBO) used in Variational Autoencoders, lacks a guaranteed monotonic relationship with the true log-likelihood of the data. This leads to an "inference gap" where optimizing the standard loss does not necessarily improve the model. This mathematical flaw often results in "prior collapse," where the model fails to learn a sufficiently rich functional prior, leading to persistent underfitting.
The SI-NP Framework
To address this, the authors reframe NP training through the lens of Variational Expectation Maximization (EM). Instead of a single-step optimization, the functional prior is treated as a latent variable that is iteratively refined.
The researchers derived a new surrogate objective that provides a rigorous improvement guarantee for the target log-likelihood. By employing Self-normalized Importance Sampling, the SI-NP model uses multiple weighted samples to estimate the EM objective. This approach allows the model to maintain a healthy level of uncertainty in its functional prior. Notably, the framework is architecture-agnostic; for instance, the popular Conditional Neural Process (CNP) is mathematically shown to be a "collapsed" special case of SI-NP when only one Monte Carlo sample is used.
Experimental Evaluation
The SI-NP was evaluated across three distinct domains: synthetic 1D regression, image completion (MNIST, CIFAR-10, SVHN), and Sim2Real transitions.
Key Results
- Superior Likelihoods: In image completion tasks, SI-NP consistently outperformed vanilla NPs. For example, on MNIST, SI-NP achieved a log-likelihood of 1.02 compared to 0.73 for the vanilla NP.
- Robust Curve Fitting: Qualitative results in 1D regression show that SI-NP provides much tighter fits to the ground truth while maintaining more reliable predictive means compared to other NP variants.
- State-of-the-Art Performance: When combined with attention mechanisms (SI-ANP), the model achieved top-tier results on complex datasets like CIFAR-10, proving that the objective function was a critical missing piece in NP optimization.
Conclusion
By replacing the flawed standard objective with an EM-based surrogate, SI-NP effectively bridges the inference gap in Neural Processes. While the use of importance sampling introduces some computational overhead, the significant gains in predictive accuracy and asymptotic behavior make it a robust solution for uncertainty-aware deep learning. Future work may focus on stabilizing learnable proposal distributions to further enhance performance.
Summary updated