Meta-LSQ: A Closed-Form Sobolev Inner Solver for Few-Shot Meta-Learning

Antonin SulcarXiv 2026 · arXiv

TL;DR

Meta-LSQ simplifies few-shot meta-learning by replacing the expensive, iterative inner loop of algorithms like MAML with a single, differentiable closed-form least-squares solve. By meta-learning only three scalar hyperparameters for a fixed random Fourier basis, it achieves 5.5x lower error on sinusoid benchmarks while training 36x faster than standard gradient-based methods.

What Problem Does This Paper Address?

Most meta-learning algorithms, such as Model-Agnostic Meta-Learning (MAML), adapt to new tasks using an "inner loop" of gradient descent. This approach is computationally expensive because the "outer loop" must differentiate through these gradient steps, requiring complex "double backpropagation." Furthermore, these methods typically meta-learn every single weight in a deep network—often thousands or millions of parameters—which can be overkill for smooth, low-dimensional problems.

The authors identify a gap: for many physical systems and smooth functions, we don't need the full expressive power of a deep network that learns from scratch. Instead, we can leverage classical linear solvers if we can find a way to make them "meta-learnable."

Methods

The core idea of Meta-LSQ is to replace the neural network backbone with a fixed random Fourier basis.

  1. Inner Solver: Instead of gradient descent, each task is fitted using a regularized least-squares (ridge regression) solve. This is a one-shot, closed-form operation.
  2. Meta-Parameters: Instead of thousands of weights, the model only meta-learns three scalars:
  3. $\sigma$ (Bandwidth): Controls the frequency/smoothness of the Fourier features.
  4. $\mu$ (Ridge): Controls regularization to prevent overfitting.
  5. $w$ (Derivative weight): Controls how much weight to give to derivative observations.
  6. Sobolev Fitting: Because the derivative of a sinusoid is just another sinusoid (shifted by $\pi/2$), the authors can incorporate derivative observations directly into the linear system. This allows the model to learn not just from data points, but from the "slopes" or sensitivities of the system.
  7. Differentiable Solve: The outer loop uses the implicit function theorem to differentiate through the closed-form solve, allowing for efficient tuning of the three scalars via standard optimizers like Adam.
Figure 2. One meta-training step. MAML adapts with gradient steps on all weights and differentiates through them; Meta-LSQ adapts with one ridge solve over frozen features, differentiates through it by eq. (8) and learns three scalars.

Datasets

The authors evaluated Meta-LSQ across three distinct domains:

  • Sinusoid Regression: The standard "toy" benchmark for meta-learning.
  • UCI Air Quality Data: A real-world dataset involving sensor calibration under environmental drift (temperature/humidity).
  • ARES Beamline Simulation (DESY): A high-stakes physics application involving the tuning of an electron beam using magnetic lenses. This task provided actual model derivatives for Sobolev fitting.

Key Insights and Ideas

  • Kernel Sharing is Key: A Gaussian-process control experiment suggested that most of Meta-LSQ's performance gains over MAML come from simply sharing a single, well-tuned kernel across all tasks rather than trying to learn a complex initialization for a deep network.
  • Slopes Stabilize the System: Adding derivative rows (Sobolev fitting) doesn't just add more data; it significantly improves the conditioning of the linear system. Even if slopes only add a few "new" independent directions, they drastically reduce the error in both value and derivative predictions.
  • Implicit Differentiation vs. Unrolling: By using a closed-form solver, the authors avoid "unrolling" gradient steps. This makes the meta-gradient exact (up to numerical precision) and eliminates the memory overhead of storing intermediate states.
Effect of derivative rows on rank and recovery

Key Results

  • Efficiency: On 10-shot sinusoid regression, Meta-LSQ trained in just 14 seconds, compared to over 8 minutes for a strong MAML baseline.
  • Accuracy: It achieved a 5.5x lower test error than MAML on sinusoids. In the DESY beam tuning simulation, Meta-LSQ with derivatives reached a median error of 1.6 $\mu$m, outperforming both reinforcement learning agents (4.4 $\mu$m) and standard Bayesian Optimization (30.7 $\mu$m).
  • Real-world Calibration: On sensor data, Meta-LSQ outperformed MAML on several targets, though it was slightly less accurate than an exact Gaussian Process, highlighting that its strength lies in tasks where derivatives or specific operator constraints are available.
Figure 4. Tuning on the trial where Meta-LSQ with derivatives ends closest to its median: the simulated screen at the best setting after n measurements, with the mean absolute error of the four beam parameters; the dashed ellipse is the target (2σ). Last column: the final beam (solid) against the target, differences magnified five times.

Limitations and Future Work

  • Expressivity Ceiling: Because the features are frozen, Meta-LSQ is limited to the span of its basis. It struggles with non-smooth functions (like triangle waves) or functions that fall outside the "smooth, low-dimensional" family it was designed for.
  • Ridge Selection: On noiseless data, standard criteria like Generalized Cross-Validation (GCV) tend to pick a ridge that is too small, leading to poor extrapolation. The authors had to implement a "ridge floor" to maintain stability in these cases.
  • Future Directions: The authors suggest that since integrals of sinusoids are also closed-form, future work could extend this to integral observations (e.g., learning from sensor readings averaged over time). There is also potential to apply this to closed-form classification using one-vs-rest schemes.

Summary updated

Abstract

Gradient-based meta-learners such as model-agnostic meta-learning (MAML) adapt to a new task with an inner loop of gradient descent that has to be differentiated through, and they meta-learn every network weight. Meta-LSQ instead fits each task with one regularised least-squares solve over a fixed random Fourier basis and meta-learns only three scalars, a bandwidth, a ridge and a derivative weight. Since the derivative of a sinusoid is again a sinusoid, the same solve can also take derivative observations, and the fitted model has exact closed-form derivatives of any order. On 10-shot sinusoid regression Meta-LSQ trains in 14 s and ends with a 5.5 times lower error than a first-order MAML baseline that trains 36 times longer, and a Gaussian-process control suggests that most of this gain comes from sharing one kernel across tasks. In simulated beam tuning at DESY's ARES facility with model derivatives it is slightly more accurate than an ALPaCA-style model and well ahead of gradient descent. Over DESY's full range of target beams, again in simulation and from measurements alone, its best setting has a median error of 7.3 $μ$m, against 14.4 for the published final beams of their reinforcement-learning agent, which has the lower median on their 300 trials of small targets. On real sensor-calibration data a learned prior mean puts it ahead of MAML on two of four targets, but an exact Gaussian process with the same change and an ALPaCA-style model stay more accurate on most targets, and it breaks down on non-smooth targets. On noiseless targets outside the span of the basis, in-sample criteria such as generalised cross-validation choose too small a ridge.
BibTeX
@misc{sulc2026meta,
  author = {Antonin Sulc},
  title = {{Meta-LSQ: A Closed-Form Sobolev Inner Solver for Few-Shot Meta-Learning}},
  year = {2026},
  eprint = {2611.24718},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24718},
}

Loading related content...

Figure