Parameter Efficient Fine Tuning without Inference Latency

This direction is merged into another direction: inference latency overhead. Its last version stays readable and citable here.

Version 1 of 120 papers20 conceptsfrozen 2026-10-04

The rise of massive foundation models has made "full fine-tuning"—updating every weight in a model—prohibitively expensive. In response, the research community has pivoted toward Parameter-Efficient Fine-Tuning (PEFT). Methods like Adapters and Prompt Tuning allow us to adapt a model to a new task by training only a tiny fraction of its parameters.

However, as Hongling Zheng and colleagues note in their survey of model-based learning, traditional PEFT comes with a "hidden tax." While these methods save on storage and training memory, they often introduce additional layers or tokens that increase inference latency. For real-time applications, this trade-off is often unacceptable. A new wave of research is now focused on achieving the best of both worlds: extreme parameter efficiency during training and zero additional cost during deployment.

The Latency Problem in Traditional PEFT

To understand why PEFT often slows down a model, we have to look at how it modifies the computation graph.

  1. Sequential Bottlenecks: Standard adapters (like those discussed by Hao Chen in Conv-Adapter) insert new layers between existing ones. Even if these layers are small, they force the hardware to perform extra sequential operations.
  2. Sequence Expansion: Prompt tuning methods, such as the DPTDR framework proposed by Benyou Wang, prepend trainable vectors to the input. This increases the effective sequence length, which is particularly costly in Transformers where attention complexity grows quadratically with sequence length.

As Wangbo Zhao argues in Dynamic Tuning, these modifications deviate from the original optimized architecture, creating a performance bottleneck in production environments.

Structural Re-parameterization and Merging

The most direct way to eliminate inference latency is to ensure that, by the time the model is deployed, the "adapter" no longer exists as a separate entity. This is achieved through structural re-parameterization.

In the GLoRA framework, Arnav Chavan and Zhiqiang Shen demonstrate that low-rank updates can be mathematically folded back into the original weight matrices after training. Similarly, Hui Chen introduced Consolidator, which uses a two-stage process to merge "grouped" connections back into the backbone. Because the final model has the exact same architecture as the original, the inference speed remains identical.

Weight Decomposition and Masking

Another strategy avoids adding new layers entirely, instead focusing on how to efficiently modify the existing weights.

Xuehai He and Jianwei Yang proposed KAdaptation, which uses a Kronecker product-based decomposition. By representing weight updates as the product of two much smaller matrices, they can adapt a Vision Transformer using only 0.09% of the parameters. Crucially, these updates are applied directly to the existing weights, meaning no new modules are added to the computation graph.

Alternatively, researchers like Kecheng Zheng suggest that we don't need to add new knowledge at all. Their method, Regularized Mask Tuning (R-AMT), learns binary masks that "unhide" task-specific pathways already present within the frozen pre-trained weights. Since masking is just a bitwise operation on existing weights, it introduces zero overhead.

Selective and Sparse Updates

If we cannot merge the updates, we can at least make them so sparse that they don't impact the overall system.

  • Information-Rich Selection: Anchun Gui and Han Xiao developed HiFi, which identifies the most "significant" attention heads using PageRank and SVD. By updating only these high-information heads, they achieve performance gains without the structural bloat of full-layer adapters.
  • Minimalist Tuning: In a surprisingly simple approach, Moein Sorkhei and Kevin Smith introduced APLA, which tunes only a few random columns in a single layer. This avoids the complexity of multi-layer adapter modules entirely.
  • Sparse Low-Rank Adaptation: Jiangning Zhang and Teng Hu developed SaRA, which repurposes "useless" small-value parameters in the original model to learn new tasks, preserving the model's original generative speed while saving significant GPU memory.

The Future of Efficiency

The shift from "parameter efficiency" to "inference efficiency" represents a maturation of the field. Early work, like Diff Pruning by Alexander M Rush and Yoon Kim, proved that we could adapt models with tiny "diff" vectors. The current generation of research, from Consolidator to GLoRA, ensures that these tiny updates don't come at the cost of the very speed that makes foundation models useful in the first place. By treating the model's architecture as a fixed constraint rather than a flexible canvas, these researchers are making massive AI models practical for real-world, low-latency deployment.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_parameter_efficient_fine_tuning_without__4304cc45,
  title = {Parameter Efficient Fine Tuning without Inference Latency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parameter-efficient-fine-tuning-without-inference-latency/rdr_4304cc453a614ddead11d4babb9297c8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_parameter_efficient_fine_tuning_without__4304cc45_v1,
  title = {Parameter Efficient Fine Tuning without Inference Latency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parameter-efficient-fine-tuning-without-inference-latency/rdr_4304cc453a614ddead11d4babb9297c8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}