Abstract
In this paper, we discuss the non-collapsibility concept and propose a new approach based on Dirichlet process mixtures to estimate the conditional effect of covariates in non-collapsible models. Using synthetic data, we evaluate the performance of our proposed method and examine its sensitivity under different settings. We also apply our method to real data on access failure among hemodialysis patients.
Introduction
In statistical modeling, researchers often aim to quantify the effect of a covariate $X$ on an outcome $Y$. However, individual outcomes are frequently influenced by unobserved or "latent" variables ($Z$). In linear models, omitting a latent variable that is independent of the treatment usually does not bias the treatment effect estimate. However, in non-linear models—specifically logistic regression and proportional hazards models—the "marginal" effect (the average effect across the population) is mathematically guaranteed to be smaller in magnitude than the "conditional" effect (the effect for a specific individual), even when the latent variable is not a confounder. This phenomenon is known as non-collapsibility.
When these hidden factors are ignored, estimates of the true effect are "shrunk" toward the null hypothesis. Standard approaches, such as Generalized Linear Mixed Models (GLMMs), attempt to account for this by assuming latent factors follow a specific distribution, typically a Gaussian (Normal) distribution. If this assumption is incorrect—for instance, if the population is bimodal—the model fails to accurately recover the true conditional effect.
The Non-Collapsibility Phenomenon
The core challenge of non-collapsibility is that marginal models produce "flatter" probability curves compared to the steeper curves of stratum-specific models. This can lead to misleading clinical conclusions, where a potent treatment appears less effective than it truly is because the model averages out heterogeneous individual risks.
As shown in the figure above, the red marginal curve exhibits a shallower slope than the black stratum-specific curves. This visualizes how the population-level average fails to capture the true individual-level response to a covariate $X$.
Methodology: Flexible Bayesian Framework
To address the limitations of rigid distributional assumptions, this paper proposes a Bayesian Hierarchical Model utilizing a Dirichlet Process Mixture (DPM) prior. Unlike standard GLMMs, the DPM acts as a flexible clustering tool, allowing the data to naturally form an arbitrary number of clusters, each with its own parameters. This allows the model to "discover" the underlying distribution of latent variables rather than assuming it.
Two primary variations were explored:
- Mean-DPM: Clusters individuals based on differing average baseline risks (intercepts).
- Sigma-DPM: Clusters individuals based on the variability (standard deviation) of their risks.
These frameworks were applied to both logistic regression for binary outcomes and Weibull proportional hazards models for survival data.
Experimental Results
Synthetic Data Performance
The proposed method was tested against three scenarios for hidden intercepts: a standard Normal distribution, a bimodal distribution, and a mixture of distributions with high variance.
In simulations where the true effect ($\beta_1$) was set to 1.000, marginal models (like GEE) significantly underestimated the effect, yielding values as low as 0.626. In contrast, the Mean-DPM model consistently recovered the true effect with an estimate of 1.001. Furthermore, the Mean Squared Error (MSE) for the DPM approach was remarkably low (0.001) compared to marginal models (0.141), demonstrating its robustness even when the latent distribution was mis-specified as bimodal.
Real-World Application: Hemodialysis Access Failure
The framework was applied to a dataset of 1,255 hemodialysis patients tracking 1,647 records of surgical "access failure." This data is characterized by repeated measures, as patients can experience multiple failures.
A standard Cox proportional hazards model suggested that a "standard fistula" access type had a lower risk of failure than a "graft." However, the DPM models—which account for individual-level heterogeneity—reversed this finding, suggesting the fistula actually had a higher conditional risk of failure. This reversal underscores the critical importance of accounting for non-collapsibility in clinical decision-making.
Conclusion
The Bayesian DPM framework provides a powerful, robust alternative for estimating conditional effects in the presence of latent heterogeneity. While the model is computationally intensive—requiring approximately 3 hours for 300 subjects via MCMC sampling—it avoids the biases inherent in marginal models and the fragility of standard GLMMs. Future work focusing on variational inference could potentially reduce these computational costs, making the framework more accessible for large-scale clinical datasets.
Summary updated