AdaKerNet: Neural Kernel Decoding for Task-Adaptive Prediction with Multimodal Large Models

Konstantinos D. Polyzos, Eleni Oikonomou, Tara JavidiarXiv 2026 · arXiv

Adapting Multimodal Large Language Models (MLLMs) to specific downstream tasks is a bit like trying to tune a high-performance engine with only a handful of tools and very little fuel. When you have limited labeled data—the "scarce-label regime"—standard approaches like fine-tuning the entire model are often impossible (due to closed-source APIs) or prone to massive overfitting.

A new paper from researchers at UC San Diego introduces AdaKerNet, a task-adaptive neural kernel decoder designed to bridge this gap. By combining the structural reliability of kernel methods with the flexibility of neural networks, AdaKerNet allows frozen MLLMs to excel at continuous prediction tasks even when supervision is extremely limited.

TL;DR

AdaKerNet is a novel, MLLM-agnostic decoder that enables accurate prediction from frozen multimodal representations in data-scarce environments. By jointly learning a Lipschitz-controlled feature map and a task-adaptive kernel deformation, it reduces prediction error by up to 41% compared to standard neural and kernel-based baselines.

What Problem Does This Paper Address?

Foundation models like LLaVA or Gemini are incredibly powerful, but adapting them to niche tasks (like predicting the age of a patient from a skin lesion image or the price of a fashion item) remains a challenge.

Current strategies usually fall into two camps:

  1. Fine-tuning: Adjusting the model's internal parameters. This is computationally expensive and often impossible for proprietary models hidden behind APIs.
  2. Downstream Decoders: Training a small model (like an MLP) on top of frozen embeddings. While flexible, these often fail when data is sparse because they lack a strong "structural prior" to guide their learning.

The authors identify a specific "scarce-label" gap: how do we translate rich, frozen MLLM representations into accurate predictions when we only have a few hundred labeled examples?

Methods

AdaKerNet operates as a three-stage pipeline that sits on top of a frozen MLLM. The architecture is summarized in the figure below:

Figure 1: Overview of AdaKerNet: (1) A spectrally normalized neural network-based feature extractor transforms frozen MLLM embeddings into Lipschitz-controlled features, controlling the geometry of the subsequent decoding stage(s), and facilitating alignment with a reference kernel that encodes a suitable structural prior; (2) A task-adaptive neural kernel decoder that extracts a nonlinear neural kernel feature map to deform and adjust the prior selection of reference kernel to the task-specific supervision data; and (3) an MLP prediction head that maps the learned kernel feature to the output labels. Kernel reconstruction anchors the induced feature map to the reference kernel, while supervised prediction adaptively deforms it to capture task-relevant relationships. The adaptively learned kernel feature map serves as the basis of the jointly trained nonlinear neural (MLP) predictor.
  1. Lipschitz-controlled Feature Extraction: The raw MLLM embedding ($e$) is passed through a spectrally normalized neural network. This ensures that the mapping is "well-behaved"—small changes in the input don't lead to wild swings in the latent representation ($z$), preserving the semantic geometry of the original model.
  2. Reference Kernel Prior: The system uses a standard kernel (like RBF or Matern) as a "soft" structural prior. This encodes an initial assumption about how similar different data points are.
  3. Task-Adaptive Kernel Deformation: This is the core innovation. Instead of using a fixed kernel, AdaKerNet learns a nonlinear feature map ($\phi_\psi$) that deforms the reference kernel to better fit the specific task.
  4. Joint Optimization: The model is trained using a dual loss function:
  5. Kernel Reconstruction Loss: Forces the learned features to respect the reference kernel's structure.
  6. Supervised Prediction Loss: Uses the available labels to nudge the features toward task-relevant relationships.

This balance allows the model to "anchor" itself to a reliable structure while "stretching" that structure to capture the nuances of the target task.

Datasets

The authors evaluated AdaKerNet across six diverse multimodal benchmarks spanning healthcare, e-commerce, and affective computing:

  • PAD-UFES-20 (Age & Mole Lesion): Predicting patient age and lesion diameter from images, text, and tabular clinical data.
  • AMAZON-FASHION: Predicting product prices from images and reviews.
  • QUECHUA-VALENCE: Predicting emotional valence from audio and transcripts.
  • SPEECHOCEAN762 (Fluency & Prosodic): Predicting pronunciation scores from audio data.

These benchmarks were tested across four different MLLM backbones: BLIP-2, LLaVA-1.5, Qwen2.5-VL, and Gemini Embedding 2.

Key Insights and Ideas

  • Geometry is a Regularizer: By enforcing Lipschitz continuity, the authors prevent the decoder from "exploding" the embedding space, which is a common failure mode for MLPs trained on small datasets.
  • The "Soft Prior" Advantage: Unlike traditional kernel methods that use a fixed similarity function, AdaKerNet treats the kernel as a starting point. The neural network can "correct" the kernel if the data suggests that the initial similarity metric is poorly aligned with the task.
  • Pairwise Supervision: The kernel reconstruction loss provides a training signal based on pairs of data points. This acts as a powerful inductive bias that helps the model learn even when target labels ($y$) are scarce.

Key Results

The researchers compared AdaKerNet against five baselines, including standard MLPs, Transformers, and Kernel Ridge Regression (KRR).

Comparison of AdaKerNet against baselines across different label budgets (N).
  • Consistency: AdaKerNet achieved the highest $R^2$ (coefficient of determination) in 58 out of 60 evaluated configurations.
  • Error Reduction: It showed an average error reduction of up to 41% over baselines in the most data-constrained settings ($N=100$).
  • Robustness: The gains were consistent across all four MLLMs and all six datasets, proving that the method is truly model-agnostic.

Limitations and Future Work

While AdaKerNet is highly effective, the authors note a few areas for growth:

  • Computational Scaling: The kernel reconstruction loss involves pairwise comparisons, which scales quadratically ($O(N^2)$) with the batch size. While manageable for the scarce-label regime, this could become a bottleneck for very large datasets.
  • Hyperparameter Sensitivity: The performance depends on the weight ($\lambda$) balancing the two losses. The authors found that intermediate values usually work best, but finding the "sweet spot" requires some validation.
  • Task Expansion: Currently focused on continuous prediction (regression), the authors plan to extend this adaptive kernel approach to classification and other complex downstream tasks in future work.

Summary updated

Abstract

Large foundation models have been introduced with the promise of efficient adaptation to downstream tasks. Yet, under limited supervision, MLLMs, an important class of large foundation models, remain challenging to adapt to various downstream tasks. Adaptation typically relies either on MLLM parameter fine-tuning or on training neural-based decoders. Both approaches struggle under limited supervision, while fine-tuning additionally requires access to model parameters, which is often unavailable for closed-source models. We introduce AdaKerNet, a novel learnable task-adaptive neural kernel decoder. AdaKerNet is fully agnostic to the parameters of the underlying MLLM and operates solely on its (frozen) rich representations obtained from the diverse available modalities. AdaKerNet relies on (i) a set of learnable, Lipschitz-controlled multimodal features derived from these MLLM representations; (ii) a reference kernel that provides a soft structural prior on those features; and (iii) a lightweight nonlinear neural predictor that adaptively deforms that structure. Learning the kernel representation and the neural predictor jointly within a unified optimization framework allows AdaKerNet to capture features and geometric relationships relevant to the downstream task. Numerical tests across four MLLMs: BLIP-2, LLaVA-1.5, Qwen2.5-VL, and Gemini Embedding 2, and multimodal inputs spanning text, audio, images, and tabular measurements demonstrate significant and consistent improvements over direct MLP, attention-, autoencoder- and kernel-based decoders, across a range of scarce-label budgets, with average error reduction of up to 41% across baselines. These results establish AdaKerNet as an effective approach for prediction from frozen multimodal representations in the scarce label regime. Additional structural ablations highlight the complementary contributions of AdaKerNet's components.
BibTeX
@misc{polyzos2026adakernet,
  author = {Konstantinos D. Polyzos and Eleni Oikonomou and Tara Javidi},
  title = {{AdaKerNet: Neural Kernel Decoding for Task-Adaptive Prediction with Multimodal Large Models}},
  year = {2026},
  eprint = {2609.36368},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.36368},
}
Figure