Neural networks for scalar input and functional output

Sidi Wu, Cédric Beaulac, Jiguo CaoStat. Comput 2023 · DOI

TL;DR

This paper introduces a novel approach using feed-forward neural networks (NNs) to tackle the challenging problem of functional response regression with scalar inputs (FoS). By adapting NN architectures to predict functional outputs, the authors demonstrate superior performance compared to traditional methods, especially in scenarios with nonlinear relationships and a large number of predictors.

What Problem Does This Paper Address?

Functional Data Analysis (FDA) deals with data that are functions or curves, such as time series or growth trajectories. A key problem in FDA is function-on-scalar (FoS) regression, where we aim to predict a functional response based on a set of scalar predictors. This is particularly challenging when the relationship between predictors and the response is nonlinear, or when there are many predictors. Traditional methods often struggle with these complexities, either by assuming linearity or by becoming computationally intractable with high dimensionality. The authors aim to bridge this gap by leveraging the power of neural networks, which are known for their ability to model complex nonlinear relationships.

Methods

The core contribution is the adaptation of standard feed-forward neural networks for FoS regression. Instead of directly outputting a scalar value, the NNs are designed to output a finite-dimensional representation of the functional response. Two main approaches are proposed:

  • NNBB (Neural Network with Basis Coefficients Output): The NN predicts the coefficients of a basis expansion (specifically B-splines) of the functional response. The predicted functional response is then reconstructed from these coefficients.
  • NNSS (Neural Network with FPC Scores Output): The NN predicts the scores of Functional Principal Component Analysis (FPCA), which are another way to represent functional data in a lower-dimensional space.

A key innovation is the modification of the objective function to allow direct training of the NN on the functional response itself, bypassing the need for an initial estimation of basis coefficients or FPC scores. This is achieved by transforming the NN's output into a predicted functional curve and then minimizing the mean squared error (MSE) between the predicted and observed functional responses.

Additionally, the authors introduce roughness penalties to control the smoothness of the predicted functional curves, a standard practice in FDA. They propose two types: penalizing the second derivative of the predicted function and penalizing the basis coefficients directly.

Datasets

The primary dataset used for evaluation is the Age-Specific Fertility Rate (ASFR) data, which consists of fertility curves for 92 countries, along with 15 demographic and socioeconomic variables. This dataset was collected from the United Nations Gender Information (UNGEN) database. The authors also generated synthetic datasets for simulation studies.

Key Insights and Ideas

  • NNs as a powerful tool for FoS: Neural networks, with their inherent ability to model nonlinearities, are well-suited for FoS regression, especially when traditional linear models fall short.
  • Directly optimizing functional output: Modifying the objective function to directly minimize the error on the functional response (rather than intermediate representations) simplifies the training process and improves performance.
  • Basis coefficients vs. FPC scores: Both basis coefficients and FPC scores can serve as effective outputs for the NN, with FPC scores showing a slight edge in informativeness for the ASFR dataset.
  • Roughness penalties are crucial: For achieving smooth and interpretable functional outputs, incorporating roughness penalties is essential, especially when using a large number of basis functions.
  • Scalability with predictors: The proposed NN-based methods exhibit significantly better scaling with the number of predictors compared to traditional FoS regression, making them more practical for high-dimensional data.

Key Results

  • Superior predictive accuracy: In the real data application (ASFR), the proposed NN models (NNBB, NNSS, NNBR, NNSR) consistently outperformed the conventional FoS model and the FAM model in terms of Mean Squared Error of Prediction (MSEP). The NNSS model achieved the best predictive performance.
  • Effective nonlinear recovery: The NN-based models demonstrated a strong ability to recover nonlinear relationships between predictors and the functional response, as visualized in scatter plots comparing true and predicted basis coefficients.
  • Improved performance with penalized objective function: NNBR (penalized objective function) showed improved prediction error compared to non-penalized NNBR, suggesting the benefit of using a large number of basis functions with a roughness penalty.
  • Simulation study insights: Across various simulation designs (linear and nonlinear scenarios), the NN models consistently outperformed FoS, especially in nonlinear settings. NNBR often emerged as a top performer.

Limitations and Future Work

  • Hyperparameter tuning: The authors acknowledge that the large number of hyperparameters in NNs (layers, neurons, basis functions, etc.) makes grid search for tuning tedious and time-consuming.
  • Multidimensional functional data: The proposed models are currently designed for scalar inputs and univariate functional outputs. Extending them to predict multidimensional functional responses (where the functional domain is a space, e.g., R^q with q > 1) is a promising future direction.
  • Dynamic NN architectures: The authors suggest exploring more dynamic NN architectures with functional output layers, rather than relying on projecting to finite-dimensional coefficients, to better handle functional outputs.
  • Integration with SoF: Combining the proposed NN models with existing literature on scalar-on-function (SoF) regression could lead to a more comprehensive framework for analyzing various forms of functional data.

Summary updated

BibTeX
@article{wu2023neural,
  author = {Sidi Wu and Cédric Beaulac and Jiguo Cao},
  title = {{Neural networks for scalar input and functional output}},
  journal = {Stat. Comput},
  year = {2023},
  doi = {10.1007/s11222-023-10287-3},
}
Figure