Nonparametric regression with cross-classified responses

Chong Gu, Ping MaCanadian 2011 · DOI

TL;DR

This paper introduces a novel nonparametric regression framework for analyzing cross-classified response variables, extending traditional log-linear models to handle continuous covariates and complex dependencies. The authors present a suite of computational tools, including penalized likelihood estimation and cross-validation, to model these intricate relationships, demonstrating their utility with simulations and an eyetracking experiment.

What Problem Does This Paper Address?

Traditional statistical models often struggle with data where the response variable has multiple categorical dimensions that are not independent, especially when continuous covariates are involved. For instance, in linguistic studies, researchers might track eye movements (the response) across different visual stimuli (e.g., colors, objects) over time (a continuous covariate). Existing methods like log-linear models are well-suited for categorical data but become cumbersome or require parametric assumptions when continuous covariates are introduced. This paper aims to bridge this gap by developing flexible, nonparametric models that can capture complex conditional independence structures within cross-classified responses and their relationship with continuous covariates, while also accommodating potential correlations within the data.

Methods

The core of the paper lies in extending the concept of conditional density estimation using a penalized likelihood approach. The authors decompose the conditional density function into components that represent main effects and interactions, inspired by functional ANOVA. This decomposition allows for the selective inclusion or exclusion of terms to model specific conditional independence structures.

Key methodological contributions include:

  • Functional ANOVA Decomposition: The conditional density is modeled using a functional ANOVA decomposition, allowing for the exploration of interactions between different components of the response and covariates.
  • Penalized Likelihood Estimation: Nonparametric estimation is achieved via penalized likelihood, balancing goodness-of-fit with smoothness. This is implemented using reproducing kernel Hilbert spaces.
  • Cross-Validation: A crucial tool for selecting smoothing parameters, ensuring robust model performance.
  • Kullback-Leibler Projection: Used for model selection, particularly for assessing the significance of interaction terms.
  • Bayesian Confidence Intervals: Provided for log odds ratios, offering a way to quantify uncertainty in estimated relationships.
  • Random Effects: Incorporated to model correlations in longitudinal or clustered data, extending the framework to mixed-effects models.

Datasets

The methods are evaluated using:

  • Simulated Data: Limited-scale simulation studies were conducted to assess the performance of the proposed models under various conditions, including scenarios with and without random effects.
  • Eyetracking Data: A real-world dataset from linguistic studies involving participants' eye movements during visual search tasks was analyzed. This dataset consists of 288 trials, with eye fixations recorded on targets, color competitors, and object competitors over time.

Key Insights and Ideas

  • Bridging Categorical and Continuous: The framework elegantly handles situations where both categorical (cross-classified responses) and continuous (time, covariates) variables are present, a common challenge in many observational studies.
  • Flexibility in Modeling Dependencies: The functional ANOVA decomposition allows researchers to explicitly model or test for conditional independence between different aspects of the response variable, providing deeper insights into the underlying data-generating process.
  • The Power of Nonparametrics: By avoiding strong parametric assumptions, the models can capture complex, nonlinear relationships that might be missed by traditional methods.
  • Handling Correlated Data: The integration of random effects is a significant advancement, enabling the analysis of clustered or longitudinal data where observations are not independent, a frequent occurrence in experimental settings.

Key Results

  • The proposed nonparametric regression models, implemented using penalized likelihood and cross-validation, demonstrated effective performance in both simulation studies and the analysis of eyetracking data.
  • The eyetracking data analysis revealed insights into how linguistic cues (e.g., emphasized adjectives) influence visual search behavior, showing how eye fixation shifts between a target and a competitor over time.
  • The study showed that the mixed-effects models provided better performance for correlated data compared to fixed-effect models, especially when the correlation was strong.
  • The R functions developed for this framework facilitate practical application, as demonstrated by the analysis of the eyetracking data.

Limitations and Future Work

  • Computational Cost: The paper acknowledges that for multivariate covariates or a large number of ANOVA terms, the quasi-Newton optimization might become computationally infeasible, suggesting that the quasi-Newton iteration might need to be skipped in such cases.
  • "X-axis" in Log-Linear Models: The authors note that their approach offers a more flexible way to incorporate continuous covariates compared to existing surrogate log-linear models, which are primarily designed for discrete covariates.
  • Future Work: The paper suggests that further research could explore alternative treatments for continuous response variables, which would require different numerical integration techniques. The development of more efficient computational methods for complex models with many interaction terms is also an implicit area for future work.

Summary updated

Abstract

In this article, we develop regression models with cross-classified responses. Conditional independence structures can be explored/exploited through the selective inclusion/exclusion of terms in a certain functional ANOVA decomposition, and the estimation is done nonparametrically via the penalized likelihood method. A cohort of computational and data analytical tools are presented, which include cross-validation for smoothing parameter selection, Kullback–Leibler projection for model selection, and Bayesian confidence intervals for odds ratios. Random effects are introduced to model possible correlations such as those found in longitudinal and clustered data. Empirical performances of the methods are explored in simulation studies of limited scales, and a real data example is presented using some eyetracking data from linguistic studies. The techniques are implemented in a suite of R functions, whose usage is briefly described in the appendix. The Canadian Journal of Statistics 39: 591–609; 2011. © 2011 Statistical Society of Canada Dans cet article, nous développons des modèles de régression avec variables réponses provenant de classifications croisées. Les structures d'indépendance conditionnelle peuvent être explorées/exploitées grâce à l'inclusion/exclusion de termes dans une décomposition de type anova fonctionnelle et l'estimation se fait de façon non paramétrique en utilisant la méthode de vraisemblance pénalisée. Un ensemble d'outils informatiques et d'analyse de données sont présentés et incluent la validation croisée pour la sélection du paramètre de lissage, la projection de Kullback-Leibler pour la sélection de modèles et les intervalles de crédibilité bayésiens pour les rapports de cotes. Des effets aléatoires sont inclus dans le modèle pour prendre en compte les possibles corrélations telles que celles trouvées dans les données longitudinales et en grappes. Des études de simulations limitées montrent les performances empiriques de ces méthodes. Un vrai jeu de données sur l'oculométrie dans des études linguistiques est aussi traité. Ces techniques sont implantées dans un ensemble de fonctions R et nous en présentons brièvement l'utilisation en annexe. La revue canadienne de statistique 39: 591–609; 2011. © 2011 Société statistique du Canada
BibTeX
@inproceedings{gu2011nonparametric,
  author = {Chong Gu and Ping Ma},
  title = {{Nonparametric regression with cross-classified responses}},
  booktitle = {Canadian},
  year = {2011},
  doi = {10.1002/cjs.10125},
}
Figure