A coordinate network is a small neural network that takes a position, such as a pixel location $(x,y)$, and outputs the signal value there, such as a color. Left alone, these networks learn smooth, blurry versions of signals. Feeding coordinates through sines and cosines first, a step called Fourier feature mapping, fixes the blur. It also adds a dial: the frequency scale $\sigma$ of those sinusoids. If $\sigma$ is too low, edges blur. If it is too high, the network fills the gaps between observed points with high-frequency noise, called aliasing. The right setting depends on the signal and on how densely it is observed, and today it is found by sweeping $\sigma$ against held-out data, signal by signal (Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains).
We propose to make that dial a per-signal quantity that is solved exactly rather than tuned. A shared deep feature extractor is meta-learned, meaning trained across many signals so that it adapts quickly to each new one. Its output features are grouped by frequency band, and each band gets its own penalty strength. The final linear layer is fitted to each new signal's observed points by ridge regression, which is least squares with a penalty that shrinks the weights. Ridge regression has a known shortcut for its exact leave-one-out error: the error you would get by refitting with each observed point held out, computed without any refitting. Because of this shortcut, every signal can pick its own per-band penalties by minimizing an honest error estimate computed only from its own observations, inside a step that remains fully differentiable. In two sentences: the closed-form solver both fits the signal and chooses how much high frequency that signal can support. Training backpropagates through both choices so that the feature extractor learns frequency bands that are worth choosing among.
Core Claim
Fitting a linear readout in closed form on top of Fourier-mapped coordinate features is already known in one dimension. DeepTime solves the last layer of a time-index network by ridge regression on each lookback window, over fixed multi-scale "concatenated Fourier features," with a single scalar penalty (Learning Deep Time-index Models for Time Series Forecasting). Moving that recipe from time series to images is the obvious next step, and it is not our contribution. We claim something else. With sparse observations, the main failure of Fourier-feature coordinate networks is a wrong bandwidth for the particular signal. That bandwidth can be selected per signal, without a validation sweep and without a separate predictor network, by minimizing the closed-form leave-one-out error over penalties tied to frequency bands. We expect this to beat a single meta-learned bandwidth and penalty, and the gain should grow as observations become sparser and as signals differ more in their spectral content.
Mechanism
The backbone takes $B$ Fourier encodings of the coordinate at geometrically spaced scales $\sigma_1 < \dots < \sigma_B$. These pass through $B$ narrow branches, and their outputs are concatenated into a feature vector $\phi(\mathbf{v}) \in \mathbb{R}^D$ whose coordinates fall into band blocks. For one signal with $N$ observed points, let $X \in \mathbb{R}^{N\times D}$ stack the features and $Y$ the observed values. Let $\Lambda(\boldsymbol\lambda)$ be the diagonal matrix that places penalty $\lambda_b$ on every feature in band $b$. The readout and its exact leave-one-out error are
$$W(\boldsymbol\lambda) = (X^\top X + \Lambda(\boldsymbol\lambda))^{-1} X^\top Y, \qquad \mathrm{LOO}(\boldsymbol\lambda) = \frac{1}{N}\sum_{i=1}^N \left\| \frac{y_i - W(\boldsymbol\lambda)^\top x_i}{1 - h_i(\boldsymbol\lambda)} \right\|^2,$$
where $h_i = x_i^\top (X^\top X + \Lambda)^{-1} x_i$ is how strongly point $i$ influences its own fit. This identity is standard for ridge regression.
For each signal, $\boldsymbol\lambda^\star = \arg\min \mathrm{LOO}(\boldsymbol\lambda)$ is found by a few Newton steps over $B \approx 4$–$8$ scalars in log space. The outer loss is the reconstruction error on held-out query points. It is differentiated through $W(\boldsymbol\lambda^\star)$, and through $\boldsymbol\lambda^\star$ itself by the implicit function theorem, which is cheap because the inner problem has only $B$ variables.
A large penalty on a band switches that band off for that signal. Meta-training therefore pushes the backbone to make each band's features useful exactly when the signal has content at that scale.
The method uses the primal form, which inverts a $D \times D$ matrix, rather than the Woodbury dual, which inverts an $N \times N$ matrix as in R2-D2 (Meta-learning with differentiable closed-form solvers). With $D \approx 1024$ and $N$ in the tens of thousands, forming $X^\top X$ costs $O(ND^2)$ and each Cholesky factorization costs $O(D^3)$. This is the regime where the dual becomes the bottleneck the research file flags, and the primal sidesteps it.
Relation to Prior Work
Several papers supply the method. R2-D2 established that backpropagating through a ridge solver trains good features, and that freezing a pre-trained extractor under an un-meta-learned solver costs 12–14% accuracy (Meta-learning with differentiable closed-form solvers, reproduced in Reproducing Meta-learning with differentiable closed-form solvers). It used a scalar penalty fixed by meta-training. We instead choose a vector of penalties per task, using an error estimate computed only on the support set. How Important is the Train-Validation Split in Meta-Learning? motivates trusting closed-form, regularized adaptation in the sparse-support regime.
The closest competitor on penalty choice is the method of Iwata et al. (Meta-learning Task-specific Regularization Weights for Few-shot Linear Regression). It trains an attention network to predict one penalty per feature from the support set, on tabular data. That predictor is learned and must generalize to new tasks. Ours is an exact optimization of an unbiased error estimate, with no extra parameters. Our penalties are organized by frequency band, which is the structure the bandwidth problem actually has.
Kernel approaches also solve adaptation in closed form. These include NTK closed-form adaptation (Meta-Learning with Neural Tangent Kernels) and variational random-feature kernels (MetaKernel: Learning Variational Random Features With Limited Labels, Learning to Learn Kernels with Variational Random Features). They operate on image-classification embeddings, not coordinates, and they do not address per-signal bandwidth.
Meta-learners for implicit neural representations motivate the problem. Meta-SparseINR uses a gradient-based inner loop (Meta-Learning Sparse Implicit Neural Representations), and Trans-INR uses a Transformer that predicts weights (Transformers as Meta-learners for Implicit Neural Representations). Both inherit a bandwidth that is fixed at meta-training time.
Research Program
We will build the band-structured backbone and the leave-one-out inner solver, and meta-train on episodes. In each episode, a signal is observed at a random fraction $\rho$ of its coordinates, which form the support set, and is scored on the rest, which form the query set. Meta-training mixes several values of $\rho$, so that the right bandwidth genuinely varies between episodes.
The analysis has three parts. The first asks whether the selected $\boldsymbol\lambda^\star$ tracks measurable properties of each signal, namely its power spectrum and its $\rho$. The second asks whether the leave-one-out estimate ranks penalty settings the same way the true query error does. Leave-one-out error is known to be optimistic for spatially correlated points, since a held-out pixel's neighbors remain in the fit. We will therefore also test a closed-form leave-one-block-out variant, in which spatial blocks rather than single points are held out. The third asks how much of the gain comes from per-signal selection versus from band-structured features alone.
Baselines
The naive reference point is a randomly initialized Fourier-feature network fitted to each signal's support points by Adam, at the scale $\sigma$ that Tancik et al. found best overall. This is exactly the per-signal sweep practice the proposal aims to remove. For context, that paper reports 25.57 PSNR with Gaussian Fourier features versus 19.32 without them on 2D natural image regression, and 19.88 versus 15.44 on 2D CT (Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains). Those numbers come from that paper's own protocol, so they serve as a sanity anchor, not a target.
The strongest existing method is Trans-INR (Transformers as Meta-learners for Implicit Neural Representations). It is the best single-pass amortized fitter in the file, and on CelebA it reports 31.96 PSNR against 30.37 for a learned-initialization baseline. Its protocol differs from our sparse-pixel setting, so we will rerun its released architecture on our exact episodes, with the same training images, the same budget of GPU hours, and the same hyperparameter search budget. Beating it means higher query PSNR at matched or lower adaptation time, with a gap that widens at small $\rho$.
A gradient-based meta-learner, learned initialization plus a few fine-tuning steps in the style of Meta-SparseINR (Meta-Learning Sparse Implicit Neural Representations), runs under the same budget. It reports about 26.73 PSNR on CelebA at about 7,000 parameters.
Two ablations isolate the key ingredient. The first is a DeepTime-style 2D extension: the same multi-band backbone with a single scalar penalty meta-learned globally (Learning Deep Time-index Models for Time Series Forecasting). The second gives the same backbone per-band penalties predicted by an Iwata-style attention network (Meta-learning Task-specific Regularization Weights for Few-shot Linear Regression). An oracle that picks $\boldsymbol\lambda$ using the query set bounds what selection could achieve.
Why not just…?
Why not just do multi-scale kernel ridge regression on random Fourier features, with no deep network at all? This is the most serious threat. Leave-one-out or cross-validated bandwidth selection for kernel ridge regression is classical, and with dense observations it may match us. Our bet is that the meta-learned backbone supplies a learned prior over signals, such as faces or organ outlines, that a stationary kernel cannot express, and that this prior matters most when $\rho$ is small. We run exactly this baseline: random Fourier features at the same $B$ scales, the same $D$, and the same leave-one-out selection. If it ties us at $\rho \le 10\%$, the deep part of the proposal is unnecessary and we will report that.
Why not just meta-learn one good $\sigma$ and $\lambda$ for the whole dataset? That is the scalar-penalty ablation above. It wins if signals in a dataset share a spectrum and $\rho$ is fixed. For that reason, we evaluate both a fixed $\rho$ and mixed $\rho$, and we include Imagenette, whose images vary more in texture than CelebA faces. The per-signal method should only be credited where the gap opens.
Why not just let a hypernetwork or attention module predict the penalties? Iwata et al. show that learned per-feature penalties beat plain ridge on tabular data, with mean squared error of 0.888 versus 1.277 at three support points (Meta-learning Task-specific Regularization Weights for Few-shot Linear Regression). A predictor, however, must itself generalize, and it has no direct view of the fit's residuals. Leave-one-out error does. We expect the closed-form selection to be more robust when test signals have spectra or densities outside the meta-training range. The predicted-penalty ablation, evaluated on held-out values of $\rho$, decides this.
Proposed Experiments
The data are CelebA and Imagenette images with $\rho \in \{1, 5, 10, 25\}\%$ of pixels observed, 2D CT reconstruction from sparse projections following the Tancik et al. protocol, and 3D occupancy from sparse surface samples, scored by IoU (intersection over union). That paper reports 0.973 IoU with Fourier features versus 0.864 without.
We measure query PSNR or IoU, wall-clock adaptation time per signal, and memory use.
Three checks decide the claim. The first is that the per-signal method beats the scalar-penalty ablation by a margin that increases as $\rho$ decreases and under mixed-$\rho$ training. If the margin is flat, the claim fails. The second is that $\boldsymbol\lambda^\star$ correlates with each signal's measured high-frequency energy and with $\rho$, and that it comes within a small PSNR gap of the query oracle. The third is that the deep method beats leave-one-out-tuned multi-scale random-feature kernel ridge regression at low $\rho$. We will also report the gap between leave-one-out and leave-one-block-out selection, to show whether spatial correlation corrupts the estimate.
References
- Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
- Learning Deep Time-index Models for Time Series Forecasting
- Meta-learning with differentiable closed-form solvers
- Reproducing Meta-learning with differentiable closed-form solvers
- How Important is the Train-Validation Split in Meta-Learning?
- Meta-learning Task-specific Regularization Weights for Few-shot Linear Regression
- Meta-Learning with Neural Tangent Kernels
- MetaKernel: Learning Variational Random Features With Limited Labels
- Learning to Learn Kernels with Variational Random Features
- Meta-Learning Sparse Implicit Neural Representations
- Transformers as Meta-learners for Implicit Neural Representations