FAStEN: An Efficient Adaptive Method for Feature Selection and Estimation in High-Dimensional Functional Regressions

Tobia Boschi, Lorenzo Testa, Francesca Chiaromonte, Matthew ReimherrJ. Comput. Graph. Stat 2023

TL;DR

This paper introduces FAStEN, a novel and highly efficient method for feature selection and parameter estimation in high-dimensional functional regression. FAStEN significantly outperforms existing methods in terms of speed and selection accuracy, making it a powerful tool for analyzing complex functional data.

What Problem Does This Paper Address?

In many scientific fields, data often comes in the form of "functional" objects – think of time series, sensor readings over time, or images. When we want to model the relationship between these functional predictors and a response (which can also be functional), we run into a challenge: high dimensionality. Even with a moderate number of functional predictors, the number of parameters to estimate can explode, leading to overfitting and unreliable results. Traditional feature selection methods, like Lasso, struggle with this complexity. This paper tackles the problem of simultaneously selecting the most relevant functional features and estimating the complex relationships between them in a function-on-function regression setting, which is particularly challenging due to the continuous nature of both predictors and responses.

Methods

FAStEN combines several sophisticated techniques:

  • Functional Principal Components (FPCs): This is a standard technique in functional data analysis. FAStEN uses FPCs to represent the functional data in a lower-dimensional space, making the problem computationally tractable. The key is that it uses FPCs for both the response and the predictors.
  • Dual Augmented Lagrangian (DAL) Algorithm: This is the core of FAStEN's efficiency. DAL is an optimization technique that excels at handling sparse problems. FAStEN adapts a DAL algorithm to solve a newly defined objective function that incorporates an adaptive Elastic Net penalty. This allows for simultaneous feature selection and parameter estimation.
  • Adaptive Scheme: FAStEN introduces an adaptive weighting strategy. This means that after an initial selection, the algorithm refines the weights of the features, allowing it to improve both selection accuracy and reduce estimation bias, especially in noisy scenarios. This is a novel aspect compared to previous DAL implementations.

Datasets

The paper uses simulated data for extensive benchmarking and a real-world dataset:

  • Simulated Data: The authors generate synthetic data to rigorously test FAStEN against competitors and evaluate different algorithmic choices. These simulations cover various scenarios of signal-to-noise ratios, number of active features, and coefficient complexity.
  • AOMIC PIOP1 Study: This dataset consists of brain fMRI activation curves and heart rate variability (HRV) data from university students. FAStEN is applied to this dataset to identify brain voxels whose activation is associated with HRV, demonstrating its utility on a large-scale, real-world problem with over 55,000 functional predictors.

Key Insights and Ideas

  • Efficiency is King: The most striking insight is FAStEN's dramatic computational efficiency. It achieves speedups of up to 2500 times compared to existing methods, enabling analysis of much larger and more complex datasets than previously feasible.
  • Adaptive Weights Matter: The adaptive weighting scheme significantly improves the estimation of coefficient surfaces, particularly in challenging, noisy scenarios. This suggests that intelligently adjusting feature importance during the selection process is crucial for robust results.
  • Sparsity is the Key: The DAL algorithm's effectiveness hinges on the sparsity of the problem. FAStEN leverages this by efficiently reducing the dimensionality and computational burden, making it scalable to high-dimensional settings.
  • FPCs as a Foundation: Using FPCs for both response and predictors is a smart way to handle the functional nature of the data, creating a manageable matrix representation for optimization.

Key Results

  • Computational Speed: FAStEN is orders of magnitude faster than its competitor, FRegSigCom, especially in high-dimensional settings.
  • Selection Accuracy: FAStEN significantly outperforms FRegSigCom in terms of false positives, drastically reducing the number of irrelevant features selected.
  • Estimation Quality: While FRegSigCom performs slightly better in high signal-to-noise ratio scenarios, FAStEN shows superior performance in harder, low signal-to-noise ratio scenarios, particularly for "difficult" coefficient surfaces.
  • Real-World Application: On the AOMIC PIOP1 fMRI dataset, FAStEN successfully identified potential brain regions associated with heart rate variability, demonstrating its practical applicability.

Limitations and Future Work

  • Hyperparameter Tuning: While FAStEN offers efficient methods for tuning parameters like $\lambda_1$ and $\lambda_2$, determining the optimal number of FPCs (K and L) still requires careful consideration to balance accuracy and computational cost. The authors suggest future work on developing quantitative criteria for selecting K and L.
  • Group Structure: The paper notes that functional predictors can often exhibit group structures (e.g., voxels in the same brain region). Extending FAStEN to incorporate such group structures is a promising avenue for future research.
  • Confidence Bands: Providing valid confidence bands for the estimated coefficient surfaces in the original functional space is a non-trivial challenge that the authors plan to address in future work.
  • More Complex Penalties: Investigating more complex penalty functions and incorporating both scalar and functional features into the selection problem are also identified as future directions.

Summary updated

Abstract

Functional regression analysis is an established tool for many contemporary scientific applications. Regression problems involving large and complex data sets are ubiquitous, and feature selection is crucial for avoiding overfitting and achieving accurate predictions. We propose a new, flexible and ultra-efficient approach to perform feature selection in a sparse high dimensional function-on-function regression problem, and we show how to extend it to the scalar-on-function framework. Our method, called FAStEN, combines functional data, optimization, and machine learning techniques to perform feature selection and parameter estimation simultaneously. We exploit the properties of Functional Principal Components and the sparsity inherent to the Dual Augmented Lagrangian problem to significantly reduce computational cost, and we introduce an adaptive scheme to improve selection accuracy. In addition, we derive asymptotic oracle properties, which guarantee estimation and selection consistency for the proposed FAStEN estimator. Through an extensive simulation study, we benchmark our approach to the best existing competitors and demonstrate a massive gain in terms of CPU time and selection performance, without sacrificing the quality of the coefficients' estimation. The theoretical derivations and the simulation study provide a strong motivation for our approach. Finally, we present an application to brain fMRI data from the AOMIC PIOP1 study. Complete FAStEN code is provided at https://github.com/IBM/funGCN.
BibTeX
@article{boschi2023fasten,
  author = {Tobia Boschi and Lorenzo Testa and Francesca Chiaromonte and Matthew Reimherr},
  title = {{FAStEN: An Efficient Adaptive Method for Feature Selection and Estimation in High-Dimensional Functional Regressions}},
  journal = {J. Comput. Graph. Stat.},
  year = {2023},
}
Figure