Robustly estimating heterogeneity in factorial data using Rashomon Partitions

A.r. Venkateswaran, Anirudh Sankar, Arun G. Chandrasekhar, Tyler H. McCormickarXiv 2024 · DOI · arXiv

TL;DR

This paper introduces "Rashomon Partition Sets" (RPSs), a novel Bayesian framework to robustly estimate heterogeneity in factorial data. RPSs capture a set of high-quality statistical models that offer diverse yet data-supported explanations, providing a more nuanced understanding of complex relationships than single models or simple averaging. This approach is crucial for making reliable scientific and policy decisions when dealing with complex, multi-faceted data.

What Problem Does This Paper Address?

Researchers often face a dilemma when modeling complex data with many interacting factors (covariates). Choosing a model that's too simple can hide important variations (heterogeneity) across different groups, leading to misleading conclusions. Conversely, overly complex models risk finding spurious patterns that don't reflect reality. This is particularly challenging in factorial designs where multiple factors interact. Existing methods often rely on selecting a single "best" model or averaging across models, which can either oversimplify or fail to capture the full spectrum of plausible explanations supported by the data. The paper addresses this by providing a principled way to explore a set of high-quality, diverse models that are supported by the data, offering a more robust understanding of heterogeneity.

Methods

The core contribution is the Rashomon Partition Set (RPS). Instead of picking a single model, the RPS is a collection of models that have posterior densities close to the maximum a posteriori (MAP) model. This set is constructed through enumeration (not sampling) to ensure all highly supported models are considered, even if they offer very different scientific interpretations.

Key methodological components include:

  • Bayesian Framework: The approach is rooted in Bayesian inference, allowing for principled uncertainty quantification.
  • l₀ Prior: A novel l₀ prior is used on the complexity of models. This prior is information-theoretically minimax optimal and crucially, it does not impose strong assumptions about the correlations between effects, which is a significant advantage in complex factorial settings.
  • Hasse Diagrams: The paper leverages Hasse diagrams to represent the partial ordering of feature combinations. This geometric structure helps in defining and enumerating interpretable and unique models.
  • Permissibility Rules: To ensure scientific interpretability and computational tractability, the authors define "permissible partitions" based on geometric restrictions on the Hasse diagram, ensuring that models represent meaningful marginal effects and avoid "measure zero" events (highly coincidental patterns).
  • Enumeration Algorithm: A branch-and-bound algorithm is proposed to efficiently enumerate the RPS, providing bounds on its size.

Datasets

The paper evaluates its method on three empirical datasets:

  • Karlan and List (2007) Charitable Giving Data: This dataset explores the effect of price matching on charitable donations.
  • NHANES Telomere Length Data: This dataset investigates heterogeneity in telomere length based on factors like age, gender, race, and hours worked.
  • Banerjee et al. (2015) Microcredit Access Data: This dataset examines the impact of microcredit access on various outcomes, including household expenditure and business performance.

The paper also uses simulated data to demonstrate the method's performance and compare it with other approaches.

Key Insights and Ideas

  • Embracing Model Plurality: The core idea is that for complex data, there might not be a single "true" model, but rather a set of plausible explanations. RPSs embrace this plurality, offering a richer understanding than relying on a single model.
  • Interpretable Heterogeneity: By using Hasse diagrams and permissibility rules, the method ensures that the identified heterogeneity is scientifically interpretable, focusing on meaningful marginal effects rather than arbitrary splits.
  • Robustness to Assumptions: The l₀ prior is a key innovation, allowing for complex, arbitrary dependence structures among effects without imposing unrealistic independence assumptions, which is a common limitation in other methods.
  • Bridging Statistical Performance and Policy Decisions: The RPS provides a set of high-quality statistical models. This allows policymakers to consider additional factors (like implementation cost or equity) when choosing among these models, without sacrificing statistical rigor.
  • "Archetypes" for Theory Building: The identified robust patterns within the RPS can serve as "archetypes" for theory building, highlighting consistent findings across diverse data.

Key Results

  • Empirical Findings:
    • In charitable giving, the RPS contradicted previous findings that match ratios don't matter, showing robust effects for Democrats and suggesting nuanced impacts of gift maximums.
  • For telomere length, robust heterogeneity was found in race, and specifically for males/females within White and Black populations, challenging some existing literature.
  • In microcredit access, the RPS revealed robust patterns for large households with no prior businesses and for those with prior business experience, offering guidance for policy interventions.
  • Simulation Results: Simulations showed that the RPS can effectively recover the true data-generating process and outperform methods like Lasso and Causal Random Forests in identifying high-quality explanations, especially in the presence of correlated features.
  • Computational Efficiency: The proposed algorithm for enumerating the RPS is shown to be computationally feasible, with the size of the RPS bounded polynomially in the number of features and levels.

Limitations and Future Work

  • Computational Complexity: While the algorithm is efficient, enumerating the RPS can still be computationally intensive for very high-dimensional problems. The runtime increases exponentially with the number of features and levels.
  • Posterior Flatness: The benefits of interpretability diminish if the posterior distribution is very flat or has many very similar modes, leading to a very large RPS.
  • Future Work:
    • Extending the framework to a frequentist paradigm.
  • Developing strategies for identifying systematically underrepresented groups in randomized trials.
  • Blending PPMx's covariate-driven cohesion with RPS enumeration.
  • Adapting the approach for model selection in graphical models or discrete model averaging.
  • Exploring extensions to higher-order derivatives and sparse covariance structures between coefficients.

Summary updated

Abstract

In both observational data and randomized control trials, researchers select statistical models to articulate how the outcome of interest varies with combinations of observable covariates. Choosing a model that is too simple can obfuscate important heterogeneity in outcomes between covariate groups, while too much complexity risks identifying spurious patterns. In this paper, we propose a novel Bayesian framework for model uncertainty called Rashomon Partition Sets (RPSs). The RPS consists of all models that have posterior density close to the maximum a posteriori (MAP) model. We construct the RPS by enumeration, rather than sampling, which ensures that we explore all models models with high evidence in the data, even if they offer dramatically different substantive explanations. We use a l0 prior, which allows the allows us to capture complex heterogeneity without imposing strong assumptions about the associations between effects, showing this prior is minimax optimal from an information-theoretic perspective. We characterize the approximation error of (functions of) parameters computed conditional on being in the RPS relative to the entire posterior. We propose an algorithm to enumerate the RPS from the class of models that are interpretable and unique, then provide bounds on the size of the RPS. We give simulation evidence along with three empirical examples: price effects on charitable giving, heterogeneity in chromosomal structure, and the introduction of microfinance.
BibTeX
@misc{venkateswaran2024robustly,
  author = {A.R. Venkateswaran and Anirudh Sankar and Arun G. Chandrasekhar and Tyler H. McCormick},
  title = {{Robustly estimating heterogeneity in factorial data using Rashomon Partitions}},
  year = {2024},
  eprint = {2404.02141},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.02141},
}

Loading related content...

Figure