The Probability of Tiered Benefit: Partial Identification with Robust and Stable Inference

Johan de Aguas, Sebastian Krumscheid, Johan Pensar, Guido Biele

TL;DR

This paper introduces the Probability of Tiered Benefit (PB), a new counterfactual metric that measures how treatments affect outcomes with multiple ordered levels (like school grades or clinical stages). The authors develop a robust statistical framework to estimate the upper and lower bounds of this probability, even when data is non-regular or treatment effects are zero for certain groups, and apply it to analyze the complex impact of ADHD medication on student performance.

What Problem Does This Paper Address?

In causal inference, we often ask: "What is the probability that this person would have succeeded if they were treated, given that they actually failed without treatment?" This is known as the Probability of Necessity and Sufficiency (PNS). However, PNS is designed for binary outcomes (Success vs. Failure).

Real-world outcomes are rarely binary. Students receive grades (A through F), and patients have disease stages (I through IV). When outcomes have three or more "tiers," the math gets significantly harder.

The authors identify two major gaps:

  1. Identification: For binary outcomes, assuming "monotonicity" (the idea that treatment can only help or do nothing, never harm) allows us to calculate a single point estimate. The authors prove that for $K \geq 3$ tiers, monotonicity is not enough to get a single number; we can only ever find a range (bounds).
  2. Statistical "Kinks": Estimating these bounds involves "min" and "max" operators. If a treatment has zero effect on a subpopulation, the underlying math becomes non-differentiable (it has a "kink"). Standard statistical tools like the bootstrap or simple "plug-in" estimators fail in these scenarios, leading to unreliable confidence intervals.

Methods

The paper contributes both a theoretical framework and a new estimation algorithm:

  • Partial Identification: They derive "sharp bounds" (the tightest possible mathematical range) for the Probability of Tiered Benefit, both with and without the assumption of monotonicity.
  • Stabilized One-Step Correction (S1S) with Matrices: To handle the "kinks" in the math (non-regularity), they adapt the S1S procedure. While the original S1S used scalar weights to stabilize estimates, this paper introduces 2x2 stabilizing matrices. This allows the estimator to account for the correlation between the upper and lower bounds simultaneously.
  • Double Robustness: The proposed estimator is "doubly robust," meaning it remains consistent as long as either the treatment model (propensity score) or the outcome model is correctly specified.

Datasets

The authors evaluate their method using two distinct sources:

  1. Synthetic Simulation: A structural causal model designed specifically to induce "non-regularity" (where a subpopulation is immune to treatment), testing if the estimator can handle the mathematical "kinks" where others fail.
  2. Norwegian ADHD Registry Data: An observational dataset of 9,352 schoolchildren diagnosed with ADHD. This high-quality registry links:
  3. Prescription history (stimulant medication).
  4. National test scores in numeracy and reading (Grades 5 and 8).
  5. Socioeconomic factors (parental education, age at birth, etc.).

Key Insights and Ideas

  • The Monotonicity Myth: In binary causal analysis, monotonicity is the "holy grail" for point identification. This paper shows that for tiered outcomes, that grail is out of reach. Even if you assume the drug never harms anyone, you still only get a range of possible benefit probabilities.
  • The "Immunity" Challenge: If a drug does absolutely nothing for a specific group (e.g., children who would get an 'A' regardless of medication), it creates a statistical boundary. Standard models struggle to stay "conservative" at these boundaries, often underestimating the uncertainty.
  • Matrix vs. Scalar: By using a matrix-based stabilization, the authors ensure that the joint confidence region (the box containing both the lower and upper bound) is accurate, rather than just looking at each bound in isolation.

Key Results

  • Statistical Stability: In simulations where standard "plug-in" estimators had 0% joint coverage (meaning they missed the true range every single time), the authors' S1S method with stabilizing matrices achieved the nominal 95% coverage.
  • ADHD Case Study:
    • Pessimistic View: The minimum probability of benefit is low (around 8% for girls), meaning we can't be certain the medication will move a child to a higher mastery tier.
  • The Harm/Benefit Trade-off: For children with low prior test scores, the data suggests a high potential for benefit (up to 58%) but also a non-negligible risk of "tiered harm" (up to 46%).
  • No "Silver Bullet": Because the bounds for benefit and harm overlap significantly across almost all groups, the authors conclude that a universal recommendation for ADHD medication remains elusive based on test scores alone.

Limitations and Future Work

  • Computational Intensity: The S1S method is computationally expensive. It requires re-learning nuisance parameters (the underlying models) as more data is processed, leading to a runtime of $O(n^2 \log n)$. This makes it difficult to scale to millions of observations.
  • Unmeasured Confounding: Like all methods based on "conditional ignorability," the results rely on the assumption that all factors influencing both treatment and outcome were measured. While the Norwegian registries are comprehensive, some hidden factors (like home environment quality) might still be missing.
  • Future Directions: The authors suggest integrating "online learning" paradigms to improve computational efficiency and exploring TMLE-based (Targeted Maximum Likelihood Estimation) solutions for non-differentiable parameters.

Summary updated

Abstract

We define the Probability of Tiered Benefit in scenarios with a binary exposure and an outcome that is either categorical with $K \geq 2$ ordered tiers or continuous partitioned by $K-1$ fixed thresholds into disjoint intervals. Similarly to other pure counterfactual queries, this parameter is not $g$-identifiable without additional assumptions. We demonstrate that strong monotonicity does not suffice for point identification when $K \geq 3$ and provide sharp bounds both with and without such constraint. Inference and uncertainty quantification for these bounds are challenging tasks due to potential nonregularity induced by ambiguities in the underlying individualized optimization problems. Such ambiguities can arise from immunities or null treatment effects in subpopulations with positive probability, affecting the lower bound estimate and hindering conservative inference. To address these issues, we extend the available Stabilized One-Step Correction (S1S) procedure by incorporating stratum-specific stabilizing matrices. Through simulations, we illustrate the benefits of this approach over existing alternatives. We apply our method to estimate bounds on the probabilities of tiered benefit and harm from pharmacological treatment for ADHD upon academic achievement, employing observational data from diagnosed Norwegian schoolchildren. Our findings indicate that while girls and children with low prior test performance could have moderate chances of both benefit and harm from treatment, a clear-cut recommendation remains uncertain across all strata.
BibTeX
@inproceedings{aguas2025probability,
  author = {Johan de Aguas and Sebastian Krumscheid and Johan Pensar and Guido Biele},
  title = {{The Probability of Tiered Benefit: Partial Identification with Robust and Stable Inference}},
  booktitle = {CLeaR},
  year = {2025},
  url = {https://openreview.net/forum?id=an73Bb58GV},
}

Loading related content...

Figure