Input Conditioned Dynamic Adaptation for Foundation Models

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2294 papers310 conceptsfrozen 2026-10-04

Standard Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA, Prefix Tuning, and Prompt Tuning have become the default strategy for adapting large-scale foundation models to downstream tasks. These methods typically insert a small set of trainable parameters into a frozen backbone. However, these parameters are static: once training is complete, the same learned transformation is applied to every single input, regardless of its specific content, complexity, or context.

A growing body of research suggests that this "one-size-fits-all" approach limits a model's ability to handle the nuances of diverse data. Instead, researchers are moving toward input-conditioned adaptation, where the adapter parameters themselves are generated or modified on-the-fly for each individual sample.

The Problem with Static Weights

In traditional PEFT, the model learns a global adjustment. For example, in standard Prompt Tuning, a fixed set of "soft prompt" vectors is prepended to every input. As Jing Zhang and colleagues point out in LaViP, this input-agnostic approach ignores the unique visual nuances of individual samples. If you are adapting a model to recognize "dogs," a static prompt must represent the "average" dog, failing to account for variations in lighting, pose, or breed that might require different internal attention patterns.

This limitation is even more pronounced in multi-task or multi-modal settings. Seungmin Baek and the authors of TADFormer argue that static weights prevent models from dynamically adjusting representations to account for specific objects present in an image. Similarly, in recommendation systems, Xiaoyu Kong notes in iLoRA that static adapters suffer from "negative transfer," where learning from one user's history can actually degrade performance for another user with different behaviors.

The Mechanism of Dynamic Generation

The core idea of input-conditioned adaptation is to replace fixed parameter matrices with a generator function—often a lightweight auxiliary network like a HyperNet or a MLP—that takes the input (or a representation of it) and produces the adapter weights.

  1. HyperNetworks for Weights: In VFM-Adapter, Jiaming Liu and co-authors use a HyperNet to generate input-aware parameters for dense prediction tasks like segmentation. This allows the model to tailor its feature extraction strategy to the unique characteristics of a specific visual scene.
  2. Dynamic Kernels: Hayeon Jo and the team behind iConFormer take this further by dynamically generating convolutional kernels based on the input image, allowing the Transformer to adapt its local receptive fields to the content at hand.
  3. Instance-Specific Prompts: Rather than learning a fixed prompt, Guanglin Zhou in HCVP and Ali Shakeri in FedDPG use small auxiliary networks to generate unique prompts for every input. This ensures the steering signal is precisely aligned with the semantic context of the sample.

Mitigating Task Interference

One of the most powerful applications of this idea is in Multi-Task Learning (MTL). When a single model is trained on many tasks, different tasks often "compete" for the same parameters, leading to interference.

The MixLoRA framework addresses this by using a Conditional Mixture of LoRA. Instead of one LoRA matrix, it maintains a pool of them and uses a gating mechanism to dynamically construct a custom low-rank matrix for each input. This allows the model to use different "experts" for fine-grained text recognition versus coarse-grained spatial analysis, effectively isolating task-specific knowledge within the same architecture.

Why It Works: Intuition

The intuition behind dynamic adaptation is similar to personalized medicine. A static adapter is like a single medicine prescribed to an entire population; it works for the "average" patient but is suboptimal for many. An input-conditioned adapter is like a custom-compounded medication tailored to an individual's specific symptoms and biology.

By making the adaptation dependent on the input:

  • Precision: The model can "zoom in" on relevant features (e.g., CAVPT refocuses attention on task-relevant objects).
  • Robustness: The model can ignore domain-specific noise. Shirsha Bose demonstrates in StyLIP that conditioning prompts on visual style helps CLIP generalize to unseen domains like sketches or paintings.
  • Efficiency: Because the generator is tiny (often <3% of total parameters), the model gains massive flexibility without the computational burden of full fine-tuning. Wei Zhu shows in PARA that generating prompt-specific scaling factors can even reduce inference latency compared to standard methods.

In summary, the shift from static to dynamic PEFT represents a move toward instance-aware intelligence. By allowing foundation models to reconfigure their internal logic for every query, molecule (MoRA), or medical scan (DVPT), researchers are unlocking higher performance and better generalization with a fraction of the trainable parameters.

Papers in this version

Showing 60 of 294 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 190 papers
  2. Version 1 · 2026-10-04 · 294 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_static_prompt_limitations_15f16f78,
  title = {static prompt limitations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/static-prompt-limitations/rdr_15f16f7865e04cfab82171426878d02c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_input_conditioned_dynamic_adaptation_for_15f16f78_v1,
  title = {Input Conditioned Dynamic Adaptation for Foundation Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/input-conditioned-dynamic-adaptation-for-foundation-models/rdr_15f16f7865e04cfab82171426878d02c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}