Who Are We Missing? : A Principled Approach to Characterizing the Underrepresented Population

Harsh Parikh, Rachael K. Ross, Elizabeth A. Stuart, Kara E. Rudolph

Abstract

Randomized controlled trials (RCTs) serve as the gold standard for causal inference, yet their findings often struggle with external validity due to the "research-to-practice gap." This paper introduces ROOT (Rashomon Set of Optimal Trees), a novel framework designed to identify underrepresented subgroups within trial data. By utilizing interpretable decision trees to characterize the Rashomon set of near-optimal models, ROOT allows researchers to refine target populations and improve the precision of treatment effect estimates. We demonstrate the framework's utility through synthetic experiments and a real-world case study on opioid use disorder treatment, showing that ROOT provides a principled, transparent method for navigating the tradeoff between generalizability and statistical precision.

Introduction

While randomized controlled trials (RCTs) provide high internal validity, the populations enrolled in these trials often differ significantly from the general patient populations they are intended to inform. This discrepancy—driven by strict eligibility criteria and volunteer bias—creates a significant hurdle for clinicians and policymakers. Existing methods to bridge this gap typically rely on "selection scores" to weight trial results toward a target population. However, these scores are often generated by "black box" models that lack interpretability, making it difficult for practitioners to understand exactly which subgroups are underrepresented or why certain predictions are less reliable.

This paper addresses the need for a principled, interpretable approach to characterizing underrepresented populations. We propose the ROOT (Rashomon Set of Optimal Trees) framework, which identifies subgroups where the trial data provides insufficient evidence, allowing for a more credible and precise estimation of treatment effects in the remaining population.

The ROOT Framework

The authors propose a two-stage approach to causal generalization:

  1. Design Stage: Identify a "sufficiently represented" subpopulation using the ROOT algorithm.
  2. Analysis Stage: Estimate the treatment effect specifically for that refined, well-supported population.

The Rashomon Set of Optimal Trees

The core innovation, ROOT, treats the identification of underrepresented groups as a functional optimization problem. Rather than seeking a single "best" model, ROOT identifies a Rashomon Set—a collection of different decision trees that are all nearly optimal at minimizing the variance of the treatment effect estimate.

By restricting the model space to sparse decision trees, ROOT generates "if-then" rules that are inherently interpretable. For example, instead of a single probability score, ROOT might identify that "Hispanic women with a history of amphetamine use" are a subgroup for whom the trial data is too sparse to provide a reliable estimate. The algorithm employs a stochastic search (explore-exploit) to navigate the massive space of possible tree structures, ensuring it finds robust representations without getting trapped in local minima.

Empirical Evaluation

Synthetic Data-Generating Processes

The framework was evaluated against four distinct synthetic data-generating processes (DGPs): Community, Box, Clone, and High-dimensional. These tests were designed to challenge the algorithm's ability to handle non-linearity and high-dimensional covariates. In these experiments, ROOT consistently outperformed traditional selection-score trimming and simple randomized trees, producing estimates with significantly lower standard errors.

Case Study: Opioid Use Disorder (MOUD)

To demonstrate real-world utility, the authors applied ROOT to the START Trial (N=1,271), which compared methadone and buprenorphine for opioid use disorder (MOUD). The target population was defined using the TEDS-A Dataset (N=740,015), representing individuals seeking treatment for opioid use across the United States.

ROOT identified that Black individuals with a history of amphetamine use but no cannabis use were significantly underrepresented in the START trial. By excluding these specific, poorly-supported subgroups (representing less than 1% of the target population), the researchers were able to provide a more precise and credible treatment effect estimate for the remaining 99.3% of the real-world population.

Key Insights

The Precision-Generalizability Tradeoff

ROOT highlights a fundamental tension in clinical research: the desire to generalize findings to everyone versus the need for statistical certainty. By pruning "high-variance" units—those for whom the trial provides little reliable information—ROOT allows researchers to navigate this tradeoff transparently.

Focus on Effect Modifiers

Unlike standard methods that prune based solely on rarity, ROOT prioritizes the removal of individuals who are both rare in the trial and likely to be effect modifiers (those whose treatment response differs significantly from the average). This ensures that the refinement of the target population is driven by the goal of accurate causal inference rather than simple demographic matching.

Limitations and Future Directions

While ROOT offers a significant step forward, it faces challenges in extremely high-dimensional settings where the number of effect modifiers is large; in such cases, the "interpretable" tree can become overly complex. Additionally, the framework relies on the assumption of "S-admissibility"—that all variables differentiating the trial group from the general population have been measured. Future work is also needed to determine how statistical inference should account for the uncertainty introduced by the data-adaptive process of defining the target population.

Conclusion

ROOT provides a principled and interpretable bridge between clinical trials and real-world practice. By characterizing the "Rashomon Set" of underrepresented populations, it empowers researchers to make more informed decisions about the limits of their data, ultimately leading to more reliable and actionable evidence for patient care.

Summary updated

Abstract

Randomized controlled trials (RCTs) serve as the cornerstone for understanding causal effects, yet extending inferences to target populations presents challenges due to effect heterogeneity and underrepresentation. Our paper addresses the critical issue of identifying and characterizing underrepresented subgroups in RCTs, proposing a novel framework for refining target populations to improve generalizability. We introduce an optimization-based approach, Rashomon Set of Optimal Trees (ROOT), to characterize underrepresented groups. ROOT optimizes the target subpopulation distribution by minimizing the variance of the target average treatment effect estimate, ensuring more precise treatment effect estimations. Notably, ROOT generates interpretable characteristics of the underrepresented population, aiding researchers in effective communication. Our approach demonstrates improved precision and interpretability compared to alternatives, as illustrated with synthetic data experiments. We apply our methodology to extend inferences from the Starting Treatment with Agonist Replacement Therapies (START) trial - investigating the effectiveness of medication for opioid use disorder - to the real-world population represented by the Treatment Episode Dataset: Admissions (TEDS-A). By refining target populations using ROOT, our framework offers a systematic approach to enhance decision-making accuracy and inform future trials in diverse populations.
BibTeX
@article{parikh2025i,
  author = {Harsh Parikh and Rachael K. Ross and Elizabeth A. Stuart and Kara E. Rudolph},
  title = {{<i>Who Are We Missing?</i> : A Principled Approach to Characterizing the Underrepresented Population}},
  journal = {Journal of the American Statistical Association},
  year = {2025},
  doi = {10.1080/01621459.2025.2495319},
}
Figure