Learning Adaptive Basis Functions

35 concepts27 papers1 proposal

When modeling continuous phenomena—whether predicting a time series, solving a physical field equation, or estimating a high-dimensional probability density—we often express our target function as a linear combination of simpler building blocks, known as basis functions:

$$f(x) = \sum_{i} c_i \phi_i(x)$$

Traditionally, these basis functions $\phi_i(x)$ are fixed before looking at the data. We rely on classical mathematical families like Fourier series, wavelets, or polynomials. While mathematically elegant, fixed bases suffer from a fundamental flaw: they are blind to the actual distribution of the data. If a signal contains highly localized, irregular, or non-stationary patterns, a fixed basis will struggle, requiring an inefficiently large number of coefficients to approximate the target accurately.

To overcome this limitation, a powerful paradigm has emerged: learning adaptive, data-driven basis functions. Instead of forcing data into a rigid, predefined mathematical mold, these methods construct the basis functions dynamically from the data itself. By tailoring the coordinate system to the intrinsic geometry and statistics of the dataset, these models achieve exceptional sample efficiency and expressivity.


Data-Dependent Hypothesis Spaces

The most direct way to make a basis function adapt to data is to anchor it to the observed samples themselves. In kernel methods, functions are typically represented within a Reproducing Kernel Hilbert Space (RKHS). Instead of searching through an abstract, infinite-dimensional space, we can restrict our hypothesis space to a finite subspace defined entirely by our training data.

In a foundational theoretical assessment, Quan-Wu Xiao demonstrates in LEARNING BY NONSYMMETRIC KERNELS WITH DATA DEPENDENT SPACES AND that defining the hypothesis space as the linear span of kernel functions centered specifically at the observed training data points allows the candidate functions to adapt dynamically to the available sample distribution. This sample-dependent approach ensures that the model allocates its representational capacity precisely where data actually exists.

This mechanism provides significant practical advantages in high-dimensional regression. Guodong Liu et al. leverage this in Sparse Shrunk Additive Models, showing that deriving basis functions directly from training samples allows a predictor to adapt its functional form to the specific characteristics of the data without requiring restrictive linear assumptions.

Similarly, Tieliang Gong et al. apply this to non-independent data in Regularized Modal Regression on Markov-Dependent Observations: A Theoretical Assessment, constructing the regression search space dynamically from observed samples to increase model flexibility. By operating within these data-dependent spaces, models bypass the tedious engineering required by classical methods. As Hong Chen et al. demonstrate in Group Sparse Additive Machine, data-dependent kernel spaces offer superior nonlinear approximation capabilities compared to fixed spline-based methods, completely eliminating the need for manual parameter tuning like selecting the number and location of knots.


Spectral Decomposition and Empirical Eigenfunctions

A more sophisticated strategy derives basis functions by solving an eigenvalue problem over the data distribution. If we can find the operator that describes how data varies across a domain, its eigenfunctions form the most natural coordinate system possible for that data. This is particularly powerful when high-dimensional data lives on a low-dimensional manifold.

Rafael Izbicki et al. illustrate this intuition in High-Dimensional Density Ratio Estimation with Extensions to Approximate Likelihood Computation. By constructing a basis from the eigenfunctions of a kernel operator defined on the data distribution, their density ratio estimator adapts directly to the intrinsic low-dimensional geometry of the data, avoiding the information loss typically caused by explicit dimensionality reduction techniques like Principal Component Analysis (PCA).

Fixed Basis (e.g., Fourier)          Data-Adaptive Eigenfunctions
      ▲                                     ▲
      │   ┌─┐   ┌─┐   ┌─┐                   │       _..-─-.._
      │  ┌┘ └┐ ┌┘ └┐ ┌┘ └┐                  │     .─┘        └─.
      │ ┌┘   └┐┘   └┘┘   └┐                 │    ┌┘            └┐
      └─┴─────┴─────┴─────►                 └────┴──────────────┴─►
 Blind to data distribution.             Aligns perfectly with the
 Requires many terms for curves.         underlying data manifold.

In practice, these empirical eigenfunctions can be extracted through several spectral techniques:


Learning Rotations and Coordinate Transforms

Instead of generating entirely new functional forms from scratch, another elegant way to build an adaptive basis is to take a standard, fixed orthonormal basis and learn a data-driven transformation or rotation that aligns it with the data's true correlation structure.

At a local level, Ann B. Lee et al. introduce this concept via "Treelets" in Rejoinder of: Treelets—An adaptive multi-scale basis for spare unordered data. Treelets construct an orthonormal basis by iteratively applying local rotations to the most highly correlated pairs of variables. By replacing pairs with sum and difference variables, this method builds a hierarchical, multi-scale basis that adapts directly to the underlying covariance without requiring a predefined grid.

For global covariance estimation, Guangzhi Cao et al. show in Covariance Estimation for High Dimensional Data Vectors Using the Sparse Matrix Transform that one can generalize the fixed butterfly structure of the Fast Fourier Transform by learning a sequence of Givens rotations directly from the data. This allows the transformation matrix to adapt to arbitrary, high-dimensional correlation patterns when samples are scarce.

This geometric alignment is equally vital when functional data suffers from phase variation or misalignment. Fangyi Wang et al. demonstrate in Probabilistic size-and-shape functional mixed models that a data-driven rotation of a fixed orthonormal basis represents the geometry of functional data much more efficiently than standard empirical functional PCA. This rotation successfully disentangles timing and amplitude variations, preserving localized features in signals like electrocardiograms (ECGs) that are typically obliterated by rigid analysis.


Why It Works: Expressivity and Domain Alignment

The transition from fixed mathematical transforms to learnable bases consistently yields dramatic performance improvements across diverse applications. The core reason is domain alignment: data-driven bases embed the statistical and physical properties of the problem domain directly into the underlying coordinate vectors.

In computer vision, Jun Jiang et al. show in What is the space of spectral sensitivity functions for digital color cameras? that modeling camera spectral sensitivity using data-driven bases derived from empirical data embeds domain-specific statistical knowledge. This design choice provides robust reconstruction against measurement noise where generic Fourier or polynomial bases fail.

In time-series forecasting, Zelin Ni et al. introduce BasisFormer: Attention-based Time Series Forecasting with Learnable and Interpretable Basis, which replaces fixed Fourier transforms with learnable, data-driven vectors to decompose signals. This adaptive building-block approach achieves an 11% to 15% performance boost over previous state-of-the-art methods, proving that adaptive bases are far more effective at capturing the messy, irregular patterns characteristic of real-world time series.

Fixed Grid / Basis                          Data-Adaptive Selection
┌───┬───┬───┬───┐                           ┌───┬───┬───┬───┐
├───┼───┼───┼───┤                           ├───┼─█─┼───┼───┤
├───┼───┼───┼───┤                           ├─█─┼───┼───┼─█─┤
└───┴───┴───┴───┘                           └───┴───┴─█─┴───┘
Wastes computation uniformly.               Places basis elements only
                                            where features exist.

Furthermore, data-driven bases drastically reduce computational complexity by minimizing the number of basis elements required to achieve a target precision:

Ultimately, learning adaptive basis functions works because it shifts the burden of geometric representation from the researcher to the data. By allowing the coordinate system to bend, rotate, and concentrate around the true data distribution, models achieve a level of parsimony and accuracy that fixed mathematical structures simply cannot match.

Key Foundational Work

Methods of canonical analysis for functional data (2003) — contributed 2 concept elements to this direction.
Multimodal Data Representations with Parameterized Local Structures (2002) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_data_driven_basis_learning_ef32f70d,
  title = {data-driven basis learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/data-driven-basis-learning/rdr_ef32f70deb3b406e8bfb7a39e1b0e7fd},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_data_driven_basis_learning_ef32f70d_v1,
  title = {data-driven basis learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/data-driven-basis-learning/rdr_ef32f70deb3b406e8bfb7a39e1b0e7fd/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake