Localized Diagnostics for Statistical Model Criticism

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 144 papers65 conceptsfrozen 2026-10-04

In traditional statistics, a "goodness-of-fit" test is often a binary gatekeeper. You feed it a model and some data, and it returns a p-value. If the p-value is low, you reject the model. But as Sara Algeri notes in Exhaustive Goodness of Fit Via Smoothed Inference and Graphics, these "black box" tests tell you that a model is wrong without explaining how or where it fails.

A growing body of research is shifting the focus from global metrics to localized diagnostics. Instead of asking "Is this model correct?", researchers are asking "In which regions of the feature space does this model deviate from reality?" This shift is essential for debugging generative models, ensuring fairness across demographic slices, and validating complex simulations in the physical sciences.

The Blind Spot of Global Aggregation

The fundamental problem with global tests like the Maximum Mean Discrepancy (MMD) is that they aggregate differences across the entire distribution. As Dan Pu and colleagues explain in Kernel-based Maximum-of-difference Test for Two-sample Comparison, this "sum-of-differences" approach allows local discrepancies to cancel each other out. For example, if a model over-predicts in one region but under-predicts in another, the global average might look perfect.

To solve this, the Maximum-of-Difference (MOD) test identifies the single largest local departure between distributions. By calculating a discrepancy score for every individual observation, the test ensures that subtle shifts in covariance or local clusters aren't washed away by the global mean.

Learning Where to Look

If you want to find a needle in a haystack, you don't just weigh the whole haystack; you look for the sharpest point. Researchers like Wittawat Jitkrittum and Arthur Gretton have pioneered methods that "learn" specific test locations to maximize the power of a test.

In A Linear-Time Kernel Goodness-of-Fit Test, they introduce the Finite Set Stein Discrepancy (FSSD). Rather than evaluating the model everywhere, FSSD optimizes a small set of "informative features"—specific points in the data domain that highlight the model's flaws. This creates a diagnostic map: if the test locations cluster in a specific region, the researcher knows exactly where the model is failing to capture the data's structure, such as failing to represent sharp boundaries or local clusters. This approach has even been extended to non-Euclidean spaces, such as spheres and tori, by Wenkai Xu in Interpretable Stein Goodness-of-fit Tests on Riemannian Manifold.

Adversarial Error Hunting

In high-dimensional spaces, like those found in particle physics or image generation, manual inspection is impossible. Here, researchers use machine learning to "hunt" for errors. The intuition is simple: if a binary classifier can easily distinguish between your model's samples and real data, the model is flawed.

However, the goal isn't just to get a high classification accuracy. In How to Understand Limitations of Generative Networks, Ranit Das demonstrates that by mapping the events the classifier finds "easiest" to distinguish back to their original features, researchers can pinpoint physical failure modes, such as missing distribution tails. Similarly, Tianyu Chen and James G. Scott introduced CoLT (Conditional Localization Test) in CoLT: The conditional localization test for assessing the accuracy of neural posterior estimates, which trains a dedicated function to search for regions where a model's predictions deviate from the ground truth. This adversarial search turns the validation process into a targeted debugging tool.

Visualizing Local Failure Modes

For a practitioner, the most useful diagnostic is often a visual one. David Zhao and Ann B. Lee have developed frameworks that treat local model coverage as a regression problem. In Diagnostics for conditional density models and Bayesian inference algorithms, they show how localized probability-probability (P-P) plots can characterize the type of error—such as over-confidence or mean bias—at specific locations in a high-dimensional feature space.

This is complemented by the CD-plot (Comparison Density plot) proposed by Sara Algeri in Smoothed inference and graphics via LP modeling. By visualizing the ratio of the true data density to the model density alongside confidence bands, the CD-plot allows researchers to see exactly where the model over-estimates or under-estimates the data. If the plot spikes at zero, the model is failing to capture "zero-inflation"; if it diverges in the tails, the model is underestimating extreme events.

Why Localization Matters

The move toward localization is more than a technical refinement; it is a requirement for trustworthy AI. As Ritwik Vashistha and Arya Farahi argue in I-trustworthy Models, a model that is "right on average" can still be systematically wrong for specific demographic groups or rare environmental conditions. Whether it is detecting geographic bias in geoparsing (Rui Zhu et al., Geoparsing: Solved or Biased?) or identifying where a climate emulator breaks down (Niccolo Dalmasso and Ann B. Lee, Validation of Approximate Likelihood), localized diagnostics provide the spatial and feature-space resolution necessary to move from mere detection to meaningful correction.

Papers in this version

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_localized_diagnostics_for_statistical_mo_df805b5e,
  title = {Localized Diagnostics for Statistical Model Criticism},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/localized-diagnostics-for-statistical-model-criticism/rdr_df805b5efa004acbbd5d29f4fe9bb465},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-18},
}

This exact version (version 1, frozen text):

@misc{lacuna_localized_diagnostics_for_statistical_mo_df805b5e_v1,
  title = {Localized Diagnostics for Statistical Model Criticism},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/localized-diagnostics-for-statistical-model-criticism/rdr_df805b5efa004acbbd5d29f4fe9bb465/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-18},
}