When modeling continuous phenomena—whether predicting a time series, solving a physical field equation, or estimating a high-dimensional probability density—we often express our target function as a linear combination of simpler building blocks, known as basis functions:
$$f(x) = \sum_{i} c_i \phi_i(x)$$
Traditionally, these basis functions $\phi_i(x)$ are fixed before looking at the data. We rely on classical mathematical families like Fourier series, wavelets, or polynomials. While mathematically elegant, fixed bases suffer from a fundamental flaw: they are blind to the actual distribution of the data. If a signal contains highly localized, irregular, or non-stationary patterns, a fixed basis will struggle, requiring an inefficiently large number of coefficients to approximate the target accurately.
To overcome this limitation, a powerful paradigm has emerged: learning adaptive, data-driven basis functions. Instead of forcing data into a rigid, predefined mathematical mold, these methods construct the basis functions dynamically from the data itself. By tailoring the coordinate system to the intrinsic geometry and statistics of the dataset, these models achieve exceptional sample efficiency and expressivity.
Data-Dependent Hypothesis Spaces
The most direct way to make a basis function adapt to data is to anchor it to the observed samples themselves. In kernel methods, functions are typically represented within a Reproducing Kernel Hilbert Space (RKHS). Instead of searching through an abstract, infinite-dimensional space, we can restrict our hypothesis space to a finite subspace defined entirely by our training data.
In a foundational theoretical assessment, Quan-Wu Xiao demonstrates in LEARNING BY NONSYMMETRIC KERNELS WITH DATA DEPENDENT SPACES AND that defining the hypothesis space as the linear span of kernel functions centered specifically at the observed training data points allows the candidate functions to adapt dynamically to the available sample distribution. This sample-dependent approach ensures that the model allocates its representational capacity precisely where data actually exists.
This mechanism provides significant practical advantages in high-dimensional regression. Guodong Liu et al. leverage this in Sparse Shrunk Additive Models, showing that deriving basis functions directly from training samples allows a predictor to adapt its functional form to the specific characteristics of the data without requiring restrictive linear assumptions.
Similarly, Tieliang Gong et al. apply this to non-independent data in Regularized Modal Regression on Markov-Dependent Observations: A Theoretical Assessment, constructing the regression search space dynamically from observed samples to increase model flexibility. By operating within these data-dependent spaces, models bypass the tedious engineering required by classical methods. As Hong Chen et al. demonstrate in Group Sparse Additive Machine, data-dependent kernel spaces offer superior nonlinear approximation capabilities compared to fixed spline-based methods, completely eliminating the need for manual parameter tuning like selecting the number and location of knots.
Spectral Decomposition and Empirical Eigenfunctions
A more sophisticated strategy derives basis functions by solving an eigenvalue problem over the data distribution. If we can find the operator that describes how data varies across a domain, its eigenfunctions form the most natural coordinate system possible for that data. This is particularly powerful when high-dimensional data lives on a low-dimensional manifold.
Rafael Izbicki et al. illustrate this intuition in High-Dimensional Density Ratio Estimation with Extensions to Approximate Likelihood Computation. By constructing a basis from the eigenfunctions of a kernel operator defined on the data distribution, their density ratio estimator adapts directly to the intrinsic low-dimensional geometry of the data, avoiding the information loss typically caused by explicit dimensionality reduction techniques like Principal Component Analysis (PCA).
Fixed Basis (e.g., Fourier) Data-Adaptive Eigenfunctions
▲ ▲
│ ┌─┐ ┌─┐ ┌─┐ │ _..-─-.._
│ ┌┘ └┐ ┌┘ └┐ ┌┘ └┐ │ .─┘ └─.
│ ┌┘ └┐┘ └┘┘ └┐ │ ┌┘ └┐
└─┴─────┴─────┴─────► └────┴──────────────┴─►
Blind to data distribution. Aligns perfectly with the
Requires many terms for curves. underlying data manifold.
In practice, these empirical eigenfunctions can be extracted through several spectral techniques:
- Kernel Matrix Decomposition: In EigenGP: Gaussian Process Models with Adaptive Eigenfunctions, Hao Peng et al. represent eigenfunctions as a finite linear combination of kernel functions within an RKHS. Optimizing these alongside model hyperparameters allows the Gaussian Process to scale to large datasets and capture complex, nonstationary patterns. For functional response data, Hachem Kadri et al. show in Operator-valued Kernels for Learning from Functional Response Data that operator-valued kernels implicitly discover an optimal basis via spectral decomposition, preserving inherent smoothness without manual feature engineering.
- Empirical Integral Operators: Zheng-Chu Guo et al. demonstrate in Thresholded spectral algorithms for sparse approximations that utilizing empirical features derived from the eigenpairs of the empirical integral operator (the Gramian matrix) identifies the most natural coordinate system for a specific dataset, concentrating the signal into a small number of sparse coefficients. To handle infinite-dimensional spaces in structured prediction, Pierre Laforgue et al. show in Deep Sketched Output Kernel Regression for Structured Prediction that finding the eigenfunctions of a sketched empirical covariance operator projects the space into a manageable, data-dependent finite basis.
- Functional PCA: When analyzing continuous curves over time, Kun Chen demonstrates in Identifying Differentially Expressed Genes for Time-course Microarray Data through Functional Data Analysis that learning optimal basis functions via functional principal component analysis captures the primary modes of variation across subjects, offering a parsimonious and interpretable representation of complex temporal patterns.
Learning Rotations and Coordinate Transforms
Instead of generating entirely new functional forms from scratch, another elegant way to build an adaptive basis is to take a standard, fixed orthonormal basis and learn a data-driven transformation or rotation that aligns it with the data's true correlation structure.
At a local level, Ann B. Lee et al. introduce this concept via "Treelets" in Rejoinder of: Treelets—An adaptive multi-scale basis for spare unordered data. Treelets construct an orthonormal basis by iteratively applying local rotations to the most highly correlated pairs of variables. By replacing pairs with sum and difference variables, this method builds a hierarchical, multi-scale basis that adapts directly to the underlying covariance without requiring a predefined grid.
For global covariance estimation, Guangzhi Cao et al. show in Covariance Estimation for High Dimensional Data Vectors Using the Sparse Matrix Transform that one can generalize the fixed butterfly structure of the Fast Fourier Transform by learning a sequence of Givens rotations directly from the data. This allows the transformation matrix to adapt to arbitrary, high-dimensional correlation patterns when samples are scarce.
This geometric alignment is equally vital when functional data suffers from phase variation or misalignment. Fangyi Wang et al. demonstrate in Probabilistic size-and-shape functional mixed models that a data-driven rotation of a fixed orthonormal basis represents the geometry of functional data much more efficiently than standard empirical functional PCA. This rotation successfully disentangles timing and amplitude variations, preserving localized features in signals like electrocardiograms (ECGs) that are typically obliterated by rigid analysis.
Why It Works: Expressivity and Domain Alignment
The transition from fixed mathematical transforms to learnable bases consistently yields dramatic performance improvements across diverse applications. The core reason is domain alignment: data-driven bases embed the statistical and physical properties of the problem domain directly into the underlying coordinate vectors.
In computer vision, Jun Jiang et al. show in What is the space of spectral sensitivity functions for digital color cameras? that modeling camera spectral sensitivity using data-driven bases derived from empirical data embeds domain-specific statistical knowledge. This design choice provides robust reconstruction against measurement noise where generic Fourier or polynomial bases fail.
In time-series forecasting, Zelin Ni et al. introduce BasisFormer: Attention-based Time Series Forecasting with Learnable and Interpretable Basis, which replaces fixed Fourier transforms with learnable, data-driven vectors to decompose signals. This adaptive building-block approach achieves an 11% to 15% performance boost over previous state-of-the-art methods, proving that adaptive bases are far more effective at capturing the messy, irregular patterns characteristic of real-world time series.
Fixed Grid / Basis Data-Adaptive Selection
┌───┬───┬───┬───┐ ┌───┬───┬───┬───┐
├───┼───┼───┼───┤ ├───┼─█─┼───┼───┤
├───┼───┼───┼───┤ ├─█─┼───┼───┼─█─┤
└───┴───┴───┴───┘ └───┴───┴─█─┴───┘
Wastes computation uniformly. Places basis elements only
where features exist.
Furthermore, data-driven bases drastically reduce computational complexity by minimizing the number of basis elements required to achieve a target precision:
- Quantum Chemistry: Jianfeng Lu et al. demonstrate in Fast algorithm for periodic density fitting for Bloch waves that identifying optimal spatial grid points through randomized column selection (pivoted QR) provides a highly adaptive basis for density fitting. This data-driven selection maintains high precision across multiple physical metrics, ensuring the approximation remains robust to different potential profiles while reducing computational scaling.
- Partial Differential Equations: In A Data-Driven Approach for Multiscale Elliptic PDEs with Random Coefficients Based on Intrinsic Dimension Reduction, Hongkai Zhao et al. show that learning problem-specific basis functions directly from solution samples exploits the inherent low-dimensional structure of the solution space. This adaptive learning bypasses the need for analytical bases, significantly reducing computational costs for high-dimensional random inputs.
- Deep Learning Architectures: Pushing this concept into neural network design, David Vigouroux et al. introduce Deep Sturm–Liouville: From Sample-Based to 1D Regularization with Learnable Orthogonal Basis Functions. By solving an eigenvalue problem along defined vector fields in the input space, the network generates a task-specific orthogonal basis derived from classical physics, achieving superior sample efficiency and generalization compared to standard neural networks.
- Causal Discovery: Even when uncovering causal relationships in continuous curves, Saptarshi Roy et al. show in Directed Cyclic Graph for Causal Discovery from Multivariate Functional Data that allowing spline-based expansions to evolve adaptively during the inference process enables the representation of continuous curves to adapt to both the observed data and emerging causal dependencies.
Ultimately, learning adaptive basis functions works because it shifts the burden of geometric representation from the researcher to the data. By allowing the coordinate system to bend, rotate, and concentrate around the true data distribution, models achieve a level of parsimony and accuracy that fixed mathematical structures simply cannot match.
Key Foundational Work
Methods of canonical analysis for functional data (2003) — contributed 2 concept elements to this direction.
Multimodal Data Representations with Parameterized Local Structures (2002) — contributed 2 concept elements to this direction.