Better together? Statistical learning in models made of modules

Pierre Jacob, Lawrence M. Murray, Chris Holmes, Christian P. RobertarXiv 2017 · DOI · arXiv

Abstract

In modern applications, statisticians are faced with integrating heterogeneous data modalities relevant for an inference, prediction, or decision problem. In such circumstances, it is convenient to use a graphical model to represent the statistical dependencies, via a set of connected "modules", each relating to a specific data modality, and drawing on specific domain expertise in their development. In principle, given data, the conventional statistical update then allows for coherent uncertainty quantification and information propagation through and across the modules. However, misspecification of any module can contaminate the estimate and update of others, often in unpredictable ways. In various settings, particularly when certain modules are trusted more than others, practitioners have preferred to avoid learning with the full model in favor of approaches that restrict the information propagation between modules, for example by restricting propagation to only particular directions along the edges of the graph. In this article, we investigate why these modular approaches might be preferable to the full model in misspecified settings. We propose principled criteria to choose between modular and full-model approaches. The question arises in many applied settings, including large stochastic dynamical systems, meta-analysis, epidemiological models, air pollution models, pharmacokinetics-pharmacodynamics, and causal inference with propensity scores.

The Challenge of Feedback Contamination

In modern data science, we rarely build one giant model from scratch. Instead, we glue together "modules"—specialized sub-models for different data types (e.g., a climate model, a transport model, and a public health model). Standard Bayesian logic dictates that we should update everything jointly. In a perfect world, information flows back and forth between these modules, refining our uncertainty.

However, we live in a world of misspecification. If one module is slightly wrong, the standard Bayesian update might "force" another module's parameters into unrealistic values just to make the mathematical fit work across the entire system. This "feedback contamination" can lead to wildly overconfident and incorrect predictions. While practitioners have long used ad-hoc "two-step" methods to avoid this, there has been a lack of a formal framework to decide when and how to modularize.

A graphical representation of a two-module system where parameters (Theta) and data (Y) are interconnected.

Methodology: A Decision-Theoretic Framework

To address this, we propose a decision-theoretic framework for Modularization based on the following pillars:

  1. Cutting Feedback: We formalize the "cut distribution." Instead of the full joint posterior $\pi(\theta_1, \theta_2 | Y_1, Y_2)$, we use a version where the "upstream" parameters ($\theta_1$) are only influenced by their own data ($Y_1$), while the "downstream" parameters ($\theta_2$) still learn from both. This prevents $Y_2$ from "poisoning" $\theta_1$.
  2. Logarithmic Scoring Rule: To decide between the full model and the modular one, we use the logarithmic score (related to KL divergence). This measures how well a model predicts "new" data.
  3. Prequential Predictive Scores: Since separate test sets are often unavailable, we use a "prequential" approach—predicting the $i$-th data point using the previous $i-1$ points. This provides a data-driven way to score the "full" vs. "cut" approaches.
  4. Sequential Monte Carlo (SMC): Because "cut" distributions do not play well with standard Markov Chain Monte Carlo (MCMC) algorithms, we utilize SMC samplers to estimate the necessary normalizing constants and predictive scores.

Experimental Evaluation

The method was evaluated across four distinct scenarios:

  • Biased Data (Synthetic): A simple setup where the second data source is intentionally biased, showing how the full model gets "pulled" away from the truth while the cut model remains robust.
  • Epidemiological Study: A study relating Human Papillomavirus (HPV) prevalence (Module 1) to cervical cancer incidence (Module 2) across 13 countries.
  • Propensity Scores: A causal inference setup where researchers estimate treatment effects while accounting for confounding variables.
  • Meta-analysis: A hierarchical model involving 30 different studies, demonstrating how modularization can handle "outlier" studies that would otherwise distort global parameters.

Key Insights and Results

Our research highlights several counter-intuitive aspects of Bayesian learning under misspecification. Notably, we show that the prior can actually be a better predictor than the posterior if the data is leading the model toward a "wrong" part of the parameter space. Modularization essentially forces the model to "average" over the uncertainty of the first module rather than "conditioning" on a potentially contaminated version of it.

In all four experimental cases, the modular/cut approach yielded higher (better) predictive scores than the full joint model when misspecification was present. In the meta-analysis example, the modular approach successfully identified a "problematic" study and isolated its impact. In the propensity score example, the modular approach prevented outcome data from influencing the treatment-assignment model—a critical requirement for valid causal inference.

Limitations and Future Directions

While promising, modularization comes with a computational cost. Sampling from cut distributions via two-stage SMC requires significant overhead. Additionally, the sequential predictive score depends on the order in which data is processed, requiring averaging over permutations.

Furthermore, cutting feedback can make it harder to tell a single, unified "story" about the data, as different parts of the model effectively see different versions of the truth. Future work will focus on scaling these methods to complex graphs containing many modules and optimizing the computational efficiency of the SMC samplers.

Summary updated

Abstract

In modern applications, statisticians are faced with integrating heterogeneous data modalities relevant for an inference, prediction, or decision problem. In such circumstances, it is convenient to use a graphical model to represent the statistical dependencies, via a set of connected "modules", each relating to a specific data modality, and drawing on specific domain expertise in their development. In principle, given data, the conventional statistical update then allows for coherent uncertainty quantification and information propagation through and across the modules. However, misspecification of any module can contaminate the estimate and update of others, often in unpredictable ways. In various settings, particularly when certain modules are trusted more than others, practitioners have preferred to avoid learning with the full model in favor of approaches that restrict the information propagation between modules, for example by restricting propagation to only particular directions along the edges of the graph. In this article, we investigate why these modular approaches might be preferable to the full model in misspecified settings. We propose principled criteria to choose between modular and full-model approaches. The question arises in many applied settings, including large stochastic dynamical systems, meta-analysis, epidemiological models, air pollution models, pharmacokinetics-pharmacodynamics, and causal inference with propensity scores.
BibTeX
@misc{jacob2017better,
  author = {Pierre Jacob and Lawrence M. Murray and Chris Holmes and Christian P. Robert},
  title = {{Better together? Statistical learning in models made of modules}},
  year = {2017},
  eprint = {1708.08719},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1708.08719},
}
Figure