Partially Linear Hazard Regression for Multivariate Survival Data

Jianwen Cai, Jianqing Fan, Jiancheng Jiang, Haibo ZhouJournal of the American Statistical Association 2007 · DOI

TL;DR

This paper introduces a new statistical method for analyzing multivariate survival data, where multiple "failure" events can occur within a single subject (e.g., multiple disease recurrences in a patient). The method effectively models both linear and non-linear relationships between covariates and the risk of failure, even when the failure times are correlated within subjects, offering a more flexible and robust approach than existing techniques.

What Problem Does This Paper Address?

Analyzing survival data, which tracks the time until an event of interest occurs, is common in many fields. However, real-world scenarios often involve more complex data structures:

  • Multivariate Survival Data: Subjects can experience multiple types of failures, or multiple occurrences of the same failure type. For example, a patient might have recurrences of a disease, or individuals in a family might experience different health events. The times to these events are often correlated due to shared genetic or environmental factors.
  • Correlated Failure Times: Standard survival models often assume independence between failure times within a subject. This assumption is violated in multivariate survival data, leading to inaccurate inferences.
  • Non-linear Covariate Effects: The relationship between a predictor variable (covariate) and the risk of failure isn't always a simple linear one. For instance, the effect of cholesterol on heart disease risk might be more complex than a straight line. Existing models often assume linear effects for simplicity, potentially masking important non-linear patterns.
  • Curse of Dimensionality: When dealing with many covariates, especially in non-parametric settings, the amount of data needed to accurately estimate relationships can grow exponentially, making analysis intractable.

This paper addresses these challenges by proposing a method that can handle correlated failure times in a multivariate setting while allowing for flexible, non-linear relationships for some covariates, combined with linear relationships for others. This "partially linear" approach helps mitigate the curse of dimensionality.

Methods

The core of the paper is the development of a profile pseudo-partial likelihood estimation method. Here's a breakdown of the key ideas:

  • Marginal Hazard Model: The approach is based on modeling the "marginal hazard," which describes the instantaneous rate of failure for a specific event type, averaged over the potential dependencies within a subject. This is a common framework for multivariate survival data when the specific correlation structure isn't the primary focus.
  • Partially Linear Structure: The hazard function is modeled as: λ_ij(t) = Y_ij(t)λ_0j(t) exp[βW_ij(t) + g(Z_ij(t))] Here:

  • λ_ij(t) is the hazard rate for the j-th failure type of the i-th subject at time t.

  • Y_ij(t) is an "at-risk" indicator.
  • λ_0j(t) is an unspecified baseline hazard for the j-th failure type.
  • W_ij(t) is a vector of covariates with linear effects, captured by parameter vector β.
  • Z_ij(t) is a covariate with a potentially non-linear effect, captured by the smooth function g().
  • Pseudo-Partial Likelihood: Because failure times within a subject are correlated, a standard partial likelihood (used in Cox models) cannot be directly applied. Instead, a "pseudo-partial likelihood" is constructed, which is robust to the specific dependence structure.
  • Profile Likelihood: To estimate both the linear parameters (β) and the non-linear function (g()), a two-step process is used. First, for a fixed β, the non-linear function g() is estimated (profiled). Then, this estimated g() is used to optimize the likelihood for β.
  • Local Polynomial Fitting: The non-linear function g() is estimated using local polynomial regression. A key technical challenge is that g() is not directly estimated but rather its derivative, which is then integrated.
  • Backfitting Algorithm: An iterative algorithm is proposed to efficiently compute the estimates for both β and g().

What's novel is the combination of these elements to handle multivariate, correlated survival data with a partially linear structure, and the development of theoretical guarantees for the estimators.

Datasets

The paper uses the Framingham Heart Study dataset for its real-world application. This dataset contains longitudinal data on participants, including times to coronary heart disease (CHD) and cerebrovascular accident (CVA). The authors specifically focus on a subset of disease-free subjects and analyze the time to CHD and CVA, acknowledging that these events within individuals can be correlated.

The simulations are generated based on a multivariate extension of the Clayton and Cuzick (1985) model, allowing for controlled correlation structures and censoring levels.

Key Insights and Ideas

  • Robustness to Dependence: A significant finding is that the proposed method for estimating the non-parametric component (g()) performs well and achieves asymptotic normality regardless of the correlation structure among failure times within a subject. This is a major advantage, as it means you don't need to precisely model the complex dependencies.
  • Bridging Linear and Non-linear: The partially linear framework is powerful because it allows researchers to leverage the interpretability of linear models for some covariates while flexibly exploring non-linear effects for others, without succumbing to the "curse of dimensionality" that would arise from a fully non-parametric approach.
  • Improved Inference: The development of consistent standard error estimators using "sandwich-type" ideas is crucial. This allows for reliable hypothesis testing and confidence interval construction, which is often a bottleneck in complex statistical modeling.
  • Optimal Non-parametric Estimation: The paper shows that the non-parametric component can be estimated as efficiently as if the parametric components were known and failure times were independent, which is a strong theoretical result.

Key Results

  • Consistent Parametric Estimation: The proposed estimator for the linear parameters (β) is shown to be √n-consistent, meaning its accuracy improves with the square root of the sample size.
  • Asymptotic Normality: Both the linear (β) and non-linear (g()) components are shown to be asymptotically normal, which is essential for statistical inference.
  • Simulation Performance: Extensive simulations demonstrate that the proposed method performs well in terms of bias, standard error estimation, and coverage rates of confidence intervals, even under moderate censoring and varying degrees of within-subject correlation.
  • Real-World Application: The analysis of the Framingham Heart Study data revealed a non-linear effect of cholesterol on the risk of CHD and CVA. The estimated function showed that the risk was lowest around normal cholesterol levels (160-170 mg/dL) and increased as levels deviated from this range. This highlights the value of the non-linear modeling capability.

Limitations and Future Work

  • One-Dimensional Non-linear Covariate: The current methodology is primarily designed for a single non-linear covariate (Z). The authors suggest extending the approach to handle multivariate non-linear covariates, potentially through additive or single-index structures.
  • Variable Selection: While the paper focuses on estimation, variable selection for both linear and non-linear components is an area for future research.
  • Testing Non-parametric Significance: Developing methods to formally test the significance of the non-parametric component (g()) is identified as an important next step to provide a more complete inference toolkit.
  • Data Sparsity: In the Framingham study application, the authors note that the non-parametric estimator becomes unreliable in regions with very sparse data (e.g., cholesterol levels >360 mg/dL). This is a general limitation of non-parametric methods.

Summary updated

Abstract

This article studies estimation of partially linear hazard regression models for multivariate survival data. A profile pseudo–partial likelihood estimation method is proposed under the marginal hazard model framework. The estimation on the parameters for the linear part is accomplished by maximization of a pseudo–partial likelihood profiled over the nonparametric part. This enables us to obtain -consistent estimators of the parametric component. Asymptotic normality is obtained for the estimates of both the linear and nonlinear parts. The new technical challenge is that the nonparametric component is indirectly estimated through its integrated derivative function from a local polynomial fit. An algorithm of fast implementation of our proposed method is presented. Consistent standard error estimates using sandwich-type ideas are also developed, which facilitates inferences for the model. It is shown that the nonparametric component can be estimated as well as if the parametric components were known and the failure times within each subject were independent. Simulations are conducted to demonstrate the performance of the proposed method. A real dataset is analyzed to illustrate the proposed methodology.
BibTeX
@article{cai2007partially,
  author = {Jianwen Cai and Jianqing Fan and Jiancheng Jiang and Haibo Zhou},
  title = {{Partially Linear Hazard Regression for Multivariate Survival Data}},
  journal = {Journal of the American Statistical Association},
  year = {2007},
  doi = {10.1198/016214506000001374},
}
Figure