Indirect Statistical Inference with Guaranteed Necessity and Sufficiency

Zhigang Zhang, Xun Hu, Cheng Lü, Tommy LiuarXiv 2025 · DOI · arXiv

TL;DR

This paper introduces Necessary and Sufficient Estimation (NSE), a new statistical framework that ensures a fitted model truly preserves its distributional assumptions. By minimizing relative errors between observed and simulated order statistics, NSE provides faster convergence and greater robustness than traditional methods like Maximum Likelihood Estimation (MLE), particularly in complex scenarios like extreme value theory or biological modeling.

What Problem Does This Paper Address?

In classical statistics, we often start with an assumption: "Assume the error terms are normally distributed." We then use methods like Ordinary Least Squares (OLS) or Maximum Likelihood Estimation (MLE) to find the best parameters.

However, there is a fundamental logical gap: these methods are "sufficient" but not "necessary." They provide the best fit given the assumption is true, but they don't guarantee that the resulting residuals actually follow that assumed distribution. Even if you pass a goodness-of-fit test (like Kolmogorov-Smirnov) after the fact, the estimation process itself didn't prioritize preserving the distribution. This is a major issue in fields like AI and extreme value theory, where the tail behavior of a distribution is just as important as its mean.

Methods

The authors propose a shift from absolute errors (differences) to relative errors (quotients). Their core technique involves Maximum Ranked Quotients (MRQ).

  1. Rank and Transform: They take observed data and simulated samples from an assumed distribution, rank them (order statistics), and transform them to a unit exponential scale.
  2. The Quotient Test: They compare the ratios of these ordered samples. They prove a fundamental theorem: for two sequences following the same distribution, the "middle" and "right" segments of these ranked quotients converge to 1, while the "left" segment (near the origin) follows a specific non-degenerate distribution.
  3. Dual NS Property: They construct an optimization criterion based on these quotients. Crucially, they prove this criterion reaches its minimum if and only if the assumed and true distributions coincide. This "Necessary and Sufficient" (NS) property is rare in statistical literature.
  4. Indirect Inference: Because the method relies on comparing data to simulations, it doesn't require an explicit probability density function (PDF), making it "likelihood-free" in practice.

Datasets

The researchers validated NSE using both synthetic benchmarks and complex real-world datasets:

  • Simulated Distributions: Unit exponential, Normal, $t$-distributions (with heavy tails), and Generalized Extreme Value (GEV) distributions.
  • Motif Data: A biological dataset involving 2,155 motifs and 4,443 genes to study protein-DNA interactions and gene expression.
  • Concrete Compressive Strength: A dataset from the UCI Machine Learning Repository used to model how eight ingredients (like cement, water, and fly ash) affect the strength of concrete over time.

Key Insights and Ideas

  • Relative vs. Absolute: Traditional methods focus on absolute distance (e.g., $y - \hat{y}$). NSE focuses on relative distance ($y / \hat{y}$). Relative errors are much more sensitive to discrepancies in the "tails" of a distribution and generally converge faster.
  • The "Aha" Moment on Q-Q Plots: The authors point out that standard Q-Q plots can be visually misleading. While points might look like they sit on the diagonal (absolute distance), their relative distance might be massive, especially near the origin. NSE captures this "nonlinear similarity" that OLS misses.
  • Likelihood-Free Robustness: NSE works in "non-regular" cases where MLE fails—for example, when the range of the distribution depends on the parameters being estimated, a common headache in extreme value statistics.

Key Results

  • Preserving Normality: In linear regression simulations where the true errors were not normal (e.g., $t$-distributed), OLS residuals failed normality tests. NSE, however, adjusted the coefficients to ensure the residuals remained as close to normal as possible, passing Jarque-Bera and Lilliefors tests where OLS failed.
  • Extreme Value Superiority: In fitting Generalized Extreme Value (GEV) distributions, NSE remained stable and accurate even when the shape parameter was less than -1, a zone where MLE often becomes inefficient or fails to exist.
  • Biological Application: On the Motif dataset, even after applying standard power transformations (like the third root), OLS residuals still rejected normality. NSE successfully found a parameter set where the residuals finally satisfied the normality assumption, providing more reliable downstream inference.

Limitations and Future Work

  • Computational Complexity: The objective function used in NSE is non-convex. This means standard optimization algorithms might get stuck in local minima rather than finding the global "best" fit. The authors suggest exploring global search methods like Peak-Over-Threshold Search (POTS) in future iterations.
  • Theoretical Gaps: There are small segments in the sequence of order statistics (between the fixed-size ends and the growing middle) not yet fully covered by the current asymptotic proofs.
  • Semiparametric Expansion: The current work focuses on parametric models; extending this "Necessary and Sufficient" logic to fully nonparametric models is a suggested next step.

Summary updated

Abstract

This paper develops a new framework for indirect statistical inference with guaranteed necessity and sufficiency, applicable to continuous random variables. We prove that when comparing exponentially transformed order statistics from an assumed distribution with those from simulated unit exponential samples, the ranked quotients exhibit distinct asymptotics: the left segment converges to a non-degenerate distribution, while the middle and right segments degenerate to one. This yields a necessary and sufficient condition in probability for two sequences of continuous random variables to follow the same distribution. Building on this, we propose an optimization criterion based on relative errors between ordered samples. The criterion achieves its minimum if and only if the assumed and true distributions coincide, providing a second necessary and sufficient condition in optimization. These dual NS properties, rare in the literature, establish a fundamentally stronger inference framework than existing methods. Unlike classical approaches based on absolute errors (e.g., Kolmogorov-Smirnov), NSE exploits relative errors to ensure faster convergence, requires only mild approximability of the cumulative distribution function, and provides both point and interval estimates. Simulations and real-data applications confirm NSE's superior performance in preserving distributional assumptions where traditional methods fail.
BibTeX
@misc{zhang2025indirect,
  author = {Zhigang Zhang and Xun Hu and Cheng Lü and Tommy Liu},
  title = {{Indirect Statistical Inference with Guaranteed Necessity and Sufficiency}},
  year = {2025},
  eprint = {2509.20249},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.20249},
  howpublished = {arXiv},
}
Figure