Many prediction problems in medicine and forecasting mix two kinds of input. One is a curve, such as a brain white-matter integrity profile measured along a nerve tract or the cumulative pledge trajectory of a crowdfunding campaign. The other is a handful of ordinary numbers, such as age, education, or a cognitive test score. Kernel methods compare two patients through a single similarity score, and the right way to build that score is to ask how much each part of the curve, and each number, should count. Today that weighting is set implicitly by units: whichever input happens to have large numerical values dominates the distance. We propose to learn that weighting from the prediction task. For the curve, the weight is a smooth nonnegative function over the curve's domain. For the numbers, it is one weight per variable. Both are fit jointly with a nonlinear kernel classifier or regressor, so that "scaling" becomes "relevance-weighted comparison" rather than a preprocessing afterthought.
The opportunity is specific. Current reproducing-kernel methods for this setting enter the scalar covariates linearly, which makes their relative scale mostly harmless: separate penalties and standardization absorb it. The authors of these methods name nonlinear treatment of the scalars as the open problem. Once the scalars and the curve share one nonlinear kernel, their relative scales, and the scale at each point along the curve, directly determine what the model can learn. No per-block tuning constant can undo a bad choice there. We expect the weighting to matter most in that regime, and we design the experiments to show whether it does.
Core Claim
A reproducing kernel Hilbert space (RKHS) is a space of functions defined by a similarity kernel $K$, in which the fitted predictor is a weighted sum of kernel evaluations at training points. For a mixed observation $(X, Z)$, where $X(t)$ is a curve on a domain $\mathcal{T}$ and $Z \in \mathbb{R}^p$ holds the scalars, we define the distance
$$d_{w,s}\big((X,Z),(X',Z')\big)^2 = \int_{\mathcal{T}} w(t)\,\big(X(t)-X'(t)\big)^2\,dt \;+\; \sum_{j=1}^{p} s_j\,(Z_j - Z'_j)^2 .$$
Here $w(t) = \exp(g(t))$ is a positive weight function with $g$ restricted to be smooth. We implement this by expanding $g$ in B-splines with a second-derivative roughness penalty. The $s_j > 0$ are per-scalar weights, and the kernel is $K = \exp(-d_{w,s}^2)$. We fit $(g, s)$ jointly with the predictor by minimizing a held-out-style criterion:
$$\min_{g,\,s}\; \mathrm{LOO}(g,s) + \mu \int_{\mathcal{T}} g''(t)^2\,dt \quad \text{subject to} \quad \int_{\mathcal{T}} w(t)\,dt + \sum_j s_j = C .$$
$\mathrm{LOO}$ is the leave-one-out loss of the kernel predictor, $\mu$ controls how quickly the weight may vary along the curve, and $C$ fixes the overall bandwidth. We adopt this mass constraint from the $\ell_2$-type scale constraint of Grandvalet and Canu. The claim is that smoothness of $w$ is the ingredient that makes relevance weighting feasible for curves at realistic sample sizes. It turns hundreds of correlated grid-point weights into a few effective parameters. The resulting model is invariant to the units of both the curve and the scalars, because any unit change is absorbed into $w$ or $s_j$. It also concentrates the comparison on the curve segments and covariates that carry signal.
Relation to Prior Work
The method comes from Adaptive Scaling for Feature Selection in SVMs. That paper learns a diagonal input scaling jointly with an SVM margin and zeroed out over 1,300 irrelevant pixels on a facial expression task while keeping about 92.6% accuracy. Its scales are independent per coordinate, and it reports a 40–100× slowdown from alternating quadratic programs. We differ in two ways. First, we tie the scales along the curve through a smoothness prior, which matters when $N$ is in the low hundreds rather than the pixel-rich image setting. Second, we replace the alternating quadratic programs with a closed-form leave-one-out objective for kernel ridge models, differentiated end-to-end.
The motivating gap comes from two papers. A Reproducing Kernel Hilbert Space Framework for Functional Classification projects curves onto a smooth direction and adds scalars linearly, in a partially linear distance-weighted discrimination model. It leaves nonlinear scalar effects as future work. Functional Linear Regression with Mixed Predictors pairs a Sobolev penalty on the curve coefficient with a group Lasso on scalars and proves minimax rates. It is also linear in both blocks. Our model is nonlinear and so gives up their clean rate theory. Our method differs from both papers in that relative scale becomes a learned quantity rather than one fixed by units.
Two papers constrain the design. PIEFS: Physics-Informed Eigenfunction Features with Learnable Scaling learns an anisotropic metric inside a smoothness energy. Its learnable-metric variants underperformed the identity metric when trained without labels. We therefore never learn the weights from an unsupervised criterion; every gradient on $(g, s)$ comes from held-out prediction loss. Scale Invariant Conditional Dependence Measures shows that variance-normalized kernel dependence measures remain fooled by monotone rescalings, and fixes this by rank-transforming each variable before kernelizing. That paper supplies our strongest preprocessing competitor rather than our method.
Two further papers play smaller roles. From FD-Loss: Supervised Feature Decorrelation as a Scale-Invariant Replacement for Random Dropout, which stabilized a covariance penalty on raw tabular data by per-column $\ell_2$ normalization, we borrow one practical idea. We center and normalize each input block before learning $(g, s)$, so the learned weights are dimensionless and the optimization is well conditioned. An RKHS approach for pivotal inference in functional linear regression makes nuisance scale cancel in a test statistic. We do not use it, because our goal is prediction and relevance weighting, not invariant inference.
Research Program
We will build the weighted-distance kernel for three predictors: kernel ridge regression, a kernel least-squares classifier, and kernel distance-weighted discrimination. The first two admit an exact leave-one-out formula, so gradients with respect to the spline coefficients of $g$ and the log-weights $\log s_j$ can be computed by automatic differentiation through one $N \times N$ solve per step. This costs $O(N^3)$, which is cheap at $N$ in the hundreds. For larger $N$, Nyström low-rank approximation keeps it tractable. For distance-weighted discrimination we learn $(g, s)$ with the least-squares surrogate and then refit the final classifier. We will separately test whether a warm-started alternating refinement helps.
The curve is handled on its observation grid, with the integral computed by quadrature. For multivariate curves, such as several tract profiles, we use one weight function per curve. The hyperparameters $\mu$ and $C$ are set by an outer cross-validation over a small grid, with every method given the same number of outer fits.
The analysis has two parts. The first is a unit-invariance statement: rescaling any curve or scalar by a positive constant leaves the fitted predictor unchanged at the optimum. The second is an empirical study of recovery: whether learned $w(t)$ concentrates on planted signal regions as $N$ grows, and how the required $N$ depends on $\mu$. A rate result for the nonlinear model is out of scope. We will not claim one.
Baselines
The naive reference point is a Gaussian kernel on the raw concatenated inputs, with a single median-heuristic bandwidth. This is the model whose failure motivates the work.
The practitioner baseline standardizes everything first. Each scalar is z-scored and each curve is divided pointwise by its across-sample standard deviation. The scalar and curve blocks then get two bandwidths, tuned by grid cross-validation. We consider this the most dangerous competitor, because with only two blocks the grid search is cheap and fully captures between-block scale.
A rank baseline follows Poczos et al.: each scalar and each curve value at each $t$ is mapped through its empirical CDF before the same two-bandwidth kernel.
A capacity-matched per-coordinate baseline is Grandvalet–Canu adaptive scaling applied to the discretized curve plus scalars, with no smoothness tie across $t$. We implement it with our leave-one-out objective, so it differs from our method only in the smoothness penalty. This is the ablation that isolates the key ingredient.
A second ablation sets $w(t)$ constant while still learning the $s_j$. This measures whether along-curve weighting matters at all beyond scalar weighting.
The strongest published methods on the real data are the linear RKHS models. On ADNI, A Reproducing Kernel Hilbert Space Framework for Functional Classification reports 9.9% misclassification error for its partially linear model with scalar covariates, 11.2% for the curve-only model, and 26.3% for an SVM on scalars alone. In that paper's Simulation Scenario 1 ($N=200$), the partially linear model reaches 31.3% error. On Kickstarter, Functional Linear Regression with Mixed Predictors reports lower root mean integrated squared error (RMISE) than classical functional regression and penalized function-on-function regression. No aggregated numeric RMISE was found on Lacuna for that method, so we will reimplement it and report its number under our splits.
All methods share the same splits, preprocessing inputs, and number of outer cross-validation fits, and we report wall-clock time. Beating the field means lower error than both the reimplemented linear RKHS model and the tuned two-bandwidth standardized kernel, by more than the across-split standard error.
Why not just…?
Why not just z-score everything and tune one bandwidth per block? This fixes units and between-block scale, and on data where signal is spread evenly along the curve it may match our method. It cannot express that only one segment of a tract profile is informative. Averaging the uninformative segments into the distance dilutes the signal, and no block-level constant fixes that. The constant-$w$ ablation and the planted-segment simulations test this directly. If standardized two-bandwidth kernels match us on ADNI and Kickstarter, we will conclude that along-curve weighting does not pay off on real data of this size, and we will report that.
Why not just rank-transform, as in copula kernels? Rank transforms buy invariance to monotone distortions, which variance standardization lacks. They are still unsupervised: every variable and every point on the curve counts equally after the transform. Our simulations include monotone distortions of irrelevant covariates, where ranks help, and irrelevant high-variance curve segments, where ranks should not help. We also run our method on rank-transformed inputs, because the two ideas combine.
Why not just learn per-grid-point scales, as Grandvalet and Canu did for pixels? With 100-plus grid points and a few hundred subjects, independent weights should overfit the leave-one-out criterion. They also produce jagged relevance maps that are hard to interpret. That approach succeeded with thousands of pixels and a much richer signal. The unsmoothed ablation is exactly this alternative. Our claim fails if the ablation matches smooth $w$ at small $N$.
Why not just compress the curve with functional principal components and treat the scores as tabular features? Principal components keep high-variance directions, which need not be the predictive ones; Sang et al. use this argument to motivate direct RKHS modeling. We include principal-component scores plus learned per-score weights as a baseline, which also tests whether our gain comes from smoothness in $t$ or merely from supervised weighting of any low-dimensional summary.
Proposed Experiments
The first experiment is a controlled simulation built on Sang et al.'s Scenario 1 design. The signal lives in a known sub-interval of the curve, and the class depends nonlinearly on two scalars, for example through an interaction. We add irrelevant scalars rescaled by factors spanning $10^{-3}$ to $10^{3}$, some passed through monotone distortions, and a high-variance irrelevant curve segment. We vary $N$ from 100 to 800. We measure misclassification error, the fraction of $\int w$ that falls on the true signal interval, and runtime relative to the fixed-kernel fit. The decisive check is whether smooth $w$ beats both the two-bandwidth standardized kernel and the unsmoothed per-grid-point version at $N \le 200$. A secondary check is whether the gap closes as $N$ grows.
The second experiment uses the ADNI fractional anisotropy tract-profile setup of Sang et al., with gender, handedness, age, education, and MMSE as scalars, over repeated stratified splits with confidence intervals. The target to beat is the reported 9.9%. We also check whether learned $w(t)$ highlights anatomically coherent tract segments that are stable across splits.
The third experiment uses Kickstarter pledge-curve forecasting from Wang et al. We regress the future trajectory on the early trajectory plus campaign metadata with kernel ridge regression and measure RMISE on the forecast region against the reimplemented mixed-predictor model. For all three experiments, the finding that would change our view is standardized two-bandwidth kernels matching our method on the real datasets while losing only in simulation. That outcome would say that fine-grained relevance weighting is a small-effect refinement for present-day functional datasets, not a necessity.
References
- Adaptive Scaling for Feature Selection in SVMs
- A Reproducing Kernel Hilbert Space Framework for Functional Classification
- Functional Linear Regression with Mixed Predictors
- PIEFS: Physics-Informed Eigenfunction Features with Learnable Scaling
- Scale Invariant Conditional Dependence Measures
- FD-Loss: Supervised Feature Decorrelation as a Scale-Invariant Replacement for Random Dropout
- An Rkhs Approach for Pivotal Inference in Functional Linear Regression