Model-based regression clustering for high-dimensional data: application to functional data

Emilie DevijverAdv. Data Anal. Classif 2017 · OpenReview

TL;DR

This paper introduces two novel methods for clustering high-dimensional data where both the response and predictors can be functions. They leverage penalized regression techniques (Lasso and rank-sparsity) to handle the complexity of high-dimensional functional data, offering a robust framework for uncovering hidden structures in complex datasets.

What Problem Does This Paper Address?

In today's data-rich world, we often encounter datasets where observations are not simple numbers but complex functions (e.g., time series, spectra). Understanding relationships within these datasets, especially when both the outcome (response) and the influencing factors (predictors) are functions, is challenging. Traditional methods struggle with the high dimensionality and functional nature of such data. This paper tackles the problem of clustering these complex observations based on their underlying functional relationships, aiming to group similar functional behaviors together. This is crucial for tasks like identifying distinct patterns in customer behavior, classifying medical signals, or understanding environmental trends.

Methods

The core contribution lies in two proposed procedures for model-based regression clustering:

  • Lasso-MLE Procedure: This method first uses a Lasso-based approach to perform variable selection, identifying the most relevant functional components (represented by wavelet coefficients) for clustering. It then refits the model using Maximum Likelihood Estimation (MLE) on these selected variables. Finally, a non-asymptotic criterion (slope heuristic) is used to select the best model from a collection of candidates.
  • Lasso-Rank Procedure: This procedure builds upon the Lasso-MLE approach by additionally incorporating rank sparsity. This means it not only selects relevant variables but also considers the matrix structure of the regression coefficients, effectively reducing the dimensionality by looking for low-rank representations of the relationships. This is particularly useful when the functional relationships can be explained by a smaller number of underlying latent factors.

Both procedures extend to functional data by first projecting the functions onto a wavelet basis, transforming them into a finite-dimensional representation suitable for regression analysis.

Datasets

The authors evaluate their methods on:

  • Simulated Datasets: Four different simulated datasets are used to demonstrate the procedures' performance under varying conditions of dimensionality, signal-to-noise ratio, and cluster separation.
  • Electricity Dataset: This real-world dataset consists of half-hourly load consumption measurements for 70 days, used to cluster successive days based on their consumption patterns.
  • Tecator Dataset: This dataset comprises spectrometric data (absorbance spectra) and fat content measurements for food samples, used to cluster samples based on the relationship between their spectra and fat content.

Key Insights and Ideas

  • Wavelet Basis for Functional Data: Projecting functional data onto a wavelet basis is a powerful technique for dimensionality reduction and capturing both local and global features, making it suitable for regression and clustering.
  • Lasso for Variable Selection: The Lasso estimator is effective in identifying relevant functional components (wavelet coefficients) for clustering, especially in high-dimensional settings where many features might be irrelevant.
  • Rank Sparsity for Matrix Structure: Incorporating rank sparsity in the regression coefficients allows for modeling complex functional relationships that might be driven by a smaller number of underlying latent factors, leading to more parsimonious and interpretable models.
  • Non-Asymptotic Model Selection: The use of the slope heuristic provides a robust, non-asymptotic criterion for selecting the best model from a collection, avoiding the pitfalls of traditional asymptotic criteria like BIC or AIC in high-dimensional scenarios.
  • Unified Framework for Functional Data: The paper successfully demonstrates how existing regression clustering techniques can be adapted and extended to handle functional data by combining wavelet decomposition with penalized regression and model selection.

Key Results

  • The proposed Lasso-MLE and Lasso-Rank procedures demonstrate good clustering performance on simulated data, often outperforming traditional methods like BIC and MLE, especially in high-dimensional settings.
  • On the electricity dataset, the procedures successfully identified meaningful clusters of daily load consumption patterns, aligning with domain knowledge (e.g., distinguishing weekdays from weekends).
  • For the Tecator dataset, the methods provided interpretable clusters based on the relationship between spectral features and fat content, highlighting the practical utility of the approach.
  • The Lasso-MLE procedure generally showed better estimation accuracy, while the Lasso-Rank procedure was advantageous when a low-rank matrix structure was present in the data.

Limitations and Future Work

  • Computational Cost: The procedures, especially the Lasso-Rank, can be computationally intensive due to the construction of extensive model collections and repeated EM algorithm runs. The authors suggest strategies like selecting a subset of regularization parameters or focusing on large values to mitigate this.
  • Diagonal Covariance Matrices: The current model assumes diagonal covariance matrices within clusters, which is a simplification. Future work could explore more general covariance structures.
  • Group-Lasso Extension: The paper mentions extending the procedures using Group-Lasso for variable selection as an improvement, suggesting that this could lead to more consistent results by considering variables in groups.
  • Functional Response and Scalar Predictors: While the paper focuses on functional predictors and responses, it notes that the framework can be adapted for other combinations (e.g., functional response with scalar predictors).

Summary updated

Abstract

Finite mixture regression models are useful for modeling the relationship between response and predictors arising from different subpopulations. In this article, we study high-dimensional predictors and high-dimensional response and propose two procedures to cluster observations according to the link between predictors and the response. To reduce the dimension, we propose to use the Lasso estimator, which takes into account the sparsity and a maximum likelihood estimator penalized by the rank, to take into account the matrix structure. To choose the number of components and the sparsity level, we construct a collection of models, varying those two parameters and we select a model among this collection with a non-asymptotic criterion. We extend these procedures to functional data, where predictors and responses are functions. For this purpose, we use a wavelet-based approach. For each situation, we provide algorithms and apply and evaluate our methods both on simulated and real datasets, to understand how they work in practice.
BibTeX
@inproceedings{devijver2017model,
  author = {Emilie Devijver},
  title = {{Model-based regression clustering for high-dimensional data: application to functional data}},
  booktitle = {Adv. Data Anal. Classif},
  year = {2017},
  url = {https://openreview.net/forum?id=TZmYV-HIFj},
}
Figure