Adaptive Nonparametric Perturbations of Parametric Bayesian Models

Bo-Han Wu, Eli N. Weinstein, Sohrab Salehi, Yixin Wang, David M. BleiarXiv 2024 · DOI · arXiv

TL;DR

This paper introduces Nonparametrically Perturbed Parametric (NPP) Bayesian models—a "best of both worlds" approach that combines the data efficiency of parametric models with the robustness of nonparametric ones. By using a novel generalized Bayes approximation (gNPP) based on statistical divergences, the authors make these complex models computationally practical for real-world tasks like causal inference in genomics.

What Problem Does This Paper Address?

In Bayesian statistics, researchers usually face a difficult choice. Parametric models (like a linear regression) are "data-efficient"—they learn quickly from small datasets because they make strong assumptions about the data's shape. However, if those assumptions are wrong (model misspecification), the results can be confidently incorrect.

On the other hand, nonparametric models are incredibly flexible and can fit almost any data shape, making them robust. But they are "data-hungry" and often require massive amounts of information to provide precise answers.

The authors address a fundamental question: How can we build a model that trusts a parametric "best guess" when data is scarce, but automatically pivots to a flexible nonparametric view as more data arrives and reveals the parametric model's flaws?

Methods

The authors formalize the NPP model as a mixture of two components:

  1. The "Spike": A standard parametric Bayesian model (the "probably roughly true" guess).
  2. The "Slab": A nonparametric perturbation that can represent any distribution.

The model uses a mixing weight ($\eta_n$) that adaptively shifts between these two based on how well the parametric model explains the observed data.

The Innovation: gNPP

Standard NPP models are notoriously difficult to compute because they require calculating "marginal likelihoods," which is often impossible for complex nonparametric priors. To solve this, the authors propose generalized NPP (gNPP).

Instead of traditional Bayes factors, gNPP uses statistical divergences to measure the "mismatch" between the model and the data. Specifically, they utilize:

  • Maximum Mean Discrepancy (MMD): Measures differences in expectations within a specific function space.
  • Wasserstein Distance: Measures the "cost" of transforming one distribution into another.
  • Kernelized Stein Discrepancy (KSD): Uses the gradient of the log-density, which is useful when you don't want to sample from the model.

This allows the user to "robustify" any existing parametric model simply by having access to its posterior samples and a "backup" nonparametric model (like a Gaussian Process or BART).

Datasets

The paper evaluates the method using:

  • Synthetic Data: A Gaussian parametric model tested against "well-specified" Gaussian data and "misspecified" Skew-Normal data.
  • Single-cell RNA sequencing (scRNA-seq):
    • Primary Data: 544 T cells from an ovarian cancer patient.
  • Auxiliary Data (The "Atlas"): A massive dataset of 261,611 T cells from healthy donors used to build the initial parametric "prior" knowledge.

Key Insights and Ideas

  • The Bartlett-Lindley Effect as a Feature: In small samples, the model naturally favors the simpler parametric model even if it's slightly wrong. This is actually beneficial because, with very little data, a "wrong but simple" model often provides better estimates than a "flexible but noisy" one.
  • Adaptive Smoothing: The gNPP acts as an automated "safety valve." If you are doing causal inference and your linear model is too simple for the complex biology of a cancer cell, the gNPP detects the high divergence and shifts weight to the nonparametric component (like BART).
  • Theoretical Guarantees: The authors prove a "semiparametric Bernstein-von Mises" theorem. This essentially means that if your parametric model is right, you get the optimal $1/\sqrt{n}$ convergence rate. If it's wrong, you still converge to the truth, just at a slower nonparametric rate.

Key Results

  • Efficiency & Robustness: In simulations, the NPP model matched the fast convergence of parametric models when the data was Gaussian. When the data was Skew-Normal (misspecified), the parametric model failed to converge to the truth, but the NPP model successfully "pivoted" and found the correct values.
  • Genomic Causal Inference: The authors estimated the effect of the FOXP3 gene on other proteins (GZMH, GZMK, NKG7).
  • For NKG7, the parametric model was highly overconfident (99% probability of a positive effect).
  • The gNPP detected that the parametric model was a poor fit (low $R^2$ and non-normal residuals) and adjusted the probability down to a more cautious 77%, reflecting the true uncertainty of the biological system.

Limitations and Future Work

  • Hyperparameter Sensitivity: The "rate" parameter ($r$) in the generalized Bayes factor determines how quickly the model switches from parametric to nonparametric. While the authors provide theoretical guidance, some problem-specific tuning may still be required.
  • High-Dimensional Scaling: While MMD and KSD scale reasonably well, the Wasserstein distance becomes computationally expensive and statistically "diluted" as the number of features (dimensions) increases.
  • Computational Overhead: Although gNPP is more efficient than standard NPP, it still requires running two models (the parametric and the nonparametric "backup") and calculating divergences between them.

Overall, this work provides a rigorous yet practical framework for scientists who want to use their domain expertise (via parametric models) without being trapped by it when the data tells a different story.

Summary updated

Abstract

Parametric Bayesian modeling offers a powerful and flexible toolbox for scientific data analysis. Yet the model, however detailed, may still be wrong, and this can make inferences untrustworthy. In this paper we study nonparametrically perturbed parametric (NPP) Bayesian models, in which a parametric Bayesian model is relaxed via a distortion of its likelihood. We analyze the properties of NPP models when the target of inference is the true data distribution or some functional of it, such as in causal inference. We show that NPP models can offer the robustness of nonparametric models while retaining the data efficiency of parametric models, achieving fast convergence when the parametric model is close to true. To efficiently analyze data with an NPP model, we develop a generalized Bayes procedure to approximate its posterior. We demonstrate our method by estimating causal effects of gene expression from single cell RNA sequencing data. NPP modeling offers an efficient approach to robust Bayesian inference and can be used to robustify any parametric Bayesian model.
BibTeX
@misc{wu2024adaptive,
  author = {Bo-Han Wu and Eli N. Weinstein and Sohrab Salehi and Yixin Wang and David M. Blei},
  title = {{Adaptive Nonparametric Perturbations of Parametric Bayesian Models}},
  year = {2024},
  eprint = {2412.10683},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.10683},
}

Loading related content...

Figure