Optimal Penalized Function-on-Function Regression Under a Reproducing Kernel Hilbert Space Framework

Xiaoxiao Sun, Pang Du, Xiao Wang, Ping MaJournal of the American Statistical Association 2017 · DOI · arXiv

Abstract

Many scientific studies collect data where the response and predictor variables are both functions of time, location, or some other covariate. Understanding the relationship between these functional variables is a common goal in these studies. Motivated from two real-life examples, we present in this paper a function-on-function regression model that can be used to analyze such kind of functional data. Our estimator of the 2D coefficient function is the optimizer of a form of penalized least squares where the penalty enforces a certain level of smoothness on the estimator. Our first result is the Representer Theorem which states that the exact optimizer of the penalized least squares actually resides in a data-adaptive finite dimensional subspace although the optimization problem is defined on a function space of infinite dimensions. This theorem then allows us an easy incorporation of the Gaussian quadrature into the optimization of the penalized least squares, which can be carried out through standard numerical procedures. We also show that our estimator achieves the minimax convergence rate in mean prediction under the framework of function-on-function regression. Extensive simulation studies demonstrate the numerical advantages of our method over the existing ones, where a sparse functional data extension is also introduced. The proposed method is then applied to our motivating examples of the benchmark Canadian weather data and a histone regulation study.

TL;DR

This paper introduces a robust framework for "function-on-function" regression—where both your input and output are continuous curves—using Reproducing Kernel Hilbert Spaces (RKHS). By proving a new Representer Theorem, the authors transform a complex, infinite-dimensional optimization problem into a solvable finite-dimensional one that is mathematically guaranteed to achieve the best possible (minimax) prediction accuracy.

What Problem Does This Paper Address?

In many scientific fields, data doesn't come in neat scalar packages; it comes as functions. Think of a weather station recording temperature ($X$) and precipitation ($Y$) every minute of the day. Standard regression handles scalars, and "functional linear regression" usually handles a functional predictor to a scalar response.

The challenge arises in function-on-function regression (FoFR), where we want to predict a whole curve $Y(t)$ from another curve $X(s)$. Existing methods like Functional Principal Component Analysis (FPCA) often struggle because they truncate data into a few "important" components that might not actually be the ones driving the relationship. Other methods, like penalized B-splines, are often just approximations and lack a solid theoretical guarantee that they are "optimal." This paper seeks a method that is numerically exact, computationally efficient, and theoretically optimal.

Methods

The authors frame the problem within a Reproducing Kernel Hilbert Space (RKHS). This allows them to define a 2D coefficient function, $\beta(t, s)$, which describes how the predictor at time $s$ influences the response at time $t$.

The core technical contribution is the Representer Theorem for FoFR. In simple terms, even though the space of possible functions for $\beta$ is infinite-dimensional (and thus impossible to search through entirely), the theorem proves that the best solution actually lives in a specific, finite-dimensional subspace constructed from the observed data.

To turn this theory into a working algorithm (which they call OPFFR), they:

  1. Use Gaussian quadrature (specifically the Gauss-Legendre rule) to handle the integrals in the regression equation.
  2. Employ Generalized Cross-Validation (GCV) to automatically pick the "smoothing parameter" that prevents the model from overfitting.
  3. Introduce OPFFR-S, an extension that uses a pre-smoothing step (PACE) to handle "sparse" data where curves are only sampled at a few irregular points.

Datasets

The researchers validated their model using two distinct real-world datasets:

  • Canadian Weather Data: A classic benchmark in functional data analysis. It includes daily temperature and precipitation profiles averaged over 30 years for 35 locations in Canada.
Canadian Weather Data: Daily temperature (left) and precipitation (right) profiles for 35 locations.
  • Histone Regulation Data: A complex biological dataset looking at how histone variants (structural proteins for DNA) relate to gene expression in Arabidopsis thaliana (thale cress) during heat stress. This involved mapping ChIP-seq data (spatial) to RNA-seq data (temporal).
Histone Regulation Data: Histone levels across genomic positions (left) and gene expression levels over time (right).

Key Insights and Ideas

  • Exactness vs. Approximation: Unlike B-spline methods that provide an approximate solution to the penalized least squares problem, the Representer Theorem ensures that OPFFR finds the exact optimizer.
  • Minimax Optimality: The authors prove that their estimator achieves the "minimax convergence rate." In the world of statistics, this is the "gold standard"—it means that no other estimator can perform better in a worst-case scenario as the sample size grows.
  • The "Phase Transition": In their sparse data studies, they observed a "phase transition" where the model's performance improves dramatically once the data reaches a certain density, outperforming traditional FPCA methods which are often the default for sparse data.

Key Results

  • Simulation Superiority: In head-to-head simulations against standard FDA (B-splines) and FPCA, OPFFR consistently achieved lower Mean Integrated Squared Error (MISE), especially as the signal-to-noise ratio increased.
  • Weather Prediction: On the Canadian weather data, OPFFR achieved a median error of 21.64, significantly beating FDA (25.90) and FPCA (30.77).
  • Biological Discovery: In the histone study, the model revealed a strong negative correlation between the histone variant H3.3 and gene activity in specific genomic regions (300–350 base pairs downstream). This suggests that the loss of H3.3 is a key mechanism for turning on heat-response genes.

Limitations and Future Work

While the RKHS framework is powerful, the authors note that other approaches like Functional Partial Least Squares (PLS) or SDR (Sufficient Dimension Reduction) might be more efficient in specific cases because they incorporate response information earlier in the process. However, their initial experiments with a functional version of PLS were not promising, suggesting that further research is needed to refine those alternative directions. Additionally, the biological findings regarding H3.3 are correlative; the authors suggest future work is needed to uncover the exact physical mechanism of how these proteins organize chromatin.

Summary updated

Abstract

Many scientific studies collect data where the response and predictor variables are both functions of time, location, or some other covariate. Understanding the relationship between these functional variables is a common goal in these studies. Motivated from two real-life examples, we present in this paper a function-on-function regression model that can be used to analyze such kind of functional data. Our estimator of the 2D coefficient function is the optimizer of a form of penalized least squares where the penalty enforces a certain level of smoothness on the estimator. Our first result is the Representer Theorem which states that the exact optimizer of the penalized least squares actually resides in a data-adaptive finite dimensional subspace although the optimization problem is defined on a function space of infinite dimensions. This theorem then allows us an easy incorporation of the Gaussian quadrature into the optimization of the penalized least squares, which can be carried out through standard numerical procedures. We also show that our estimator achieves the minimax convergence rate in mean prediction under the framework of function-on-function regression. Extensive simulation studies demonstrate the numerical advantages of our method over the existing ones, where a sparse functional data extension is also introduced. The proposed method is then applied to our motivating examples of the benchmark Canadian weather data and a histone regulation study.
BibTeX
@article{sun2017optimal,
  author = {Xiaoxiao Sun and Pang Du and Xiao Wang and Ping Ma},
  title = {{Optimal Penalized Function-on-Function Regression Under a Reproducing Kernel Hilbert Space Framework}},
  journal = {Journal of the American Statistical Association},
  year = {2017},
  doi = {10.1080/01621459.2017.1356320},
}
Figure