YOLO-PEFT: Parameter-Efficient Fine-Tuning on YOLO Family

Xu Lin, WenJie Nie, Jinlong Peng, Weifu Fu, YueXiao Ma, Xiawu Zheng, Yong LiuarXiv 2026 · arXiv

Fine-tuning real-time object detectors like the YOLO family is a standard task for engineers, but applying modern Parameter-Efficient Fine-Tuning (PEFT) techniques—like LoRA—to these models is surprisingly treacherous. A new paper from researchers at Tencent and Xiamen University, "YOLO-PEFT: Parameter-Efficient Fine-Tuning on YOLO Family," introduces a structure-aware framework designed to navigate the complexities of heterogeneous detector architectures.

TL;DR

YOLO-PEFT treats adapter placement as an auditable constraint-planning problem, preventing the "silent failures" that occur when generic PEFT methods are applied to complex detector graphs. It significantly reduces training memory (by ~44%) and can even outperform full fine-tuning in accuracy, while providing a safety mechanism that refuses to train if a stable plan cannot be found.

What Problem Does This Paper Address?

In the world of Large Language Models (LLMs), PEFT is straightforward because LLMs are mostly homogeneous stacks of Transformer blocks. Real-time detectors like YOLO, however, are "heterogeneous." They interleave dense, grouped, and depthwise convolutions with specialized components like Distribution Focal Loss (DFL) projections, text-image fusion layers, and Mixture-of-Experts (MoE) routing.

The authors identify a major gap: generic PEFT interfaces select targets by name or type without understanding the underlying "operator contracts" or "detection semantics." Indiscriminately inserting LoRA into a YOLO model often leads to optimization failure or severe performance degradation. Currently, engineers rely on costly manual trial and error to find a configuration that works.

Methods

The core of the paper is the YOLO-PEFT framework, which replaces manual guesswork with a four-stage automated pipeline:

  1. Parse: A graph parser analyzes the detector and assigns "roles" to every module (e.g., backbone, neck, regression head).
  2. Plan: A structure-aware planner evaluates each module against a set of hard constraints. It checks for operator validity (e.g., can this specific convolution type support an adapter?), semantic safety (e.g., is this a sensitive DFL projection that shouldn't be touched?), and deployment compatibility.
  3. Train: The framework lowers the accepted plan into a YOLO-compatible runtime that handles sidecar adapter training while freezing the base weights.
  4. Deploy: It provides a verified path to merge adapters back into the base model and export them to production formats like ONNX or TensorRT.

![Fig.

  1. System overview showing the parsing, planning, training, and deployment stages.](/figures/art_8aaf1457efc940148ae84fff76ae2ae1)

A standout feature is the Refuse outcome. If the planner determines that no reliable plan exists within the resource budget or that the risk of "catastrophic degradation" is too high, it returns a refusal code before training even begins, saving significant GPU time.

Datasets

The researchers evaluated their framework using the official VOC07+12 trainval set for training and the VOC2007 test set for evaluation. The study covered a wide range of architectures, including:

  • CNN-based: YOLOv8, YOLO11.
  • Attention-augmented: YOLO12.
  • Multimodal: YOLO-World.
  • Transformer-based: RT-DETR.
  • MoE-based: YOLO-Master-EsMoE.

Key Insights and Ideas

  • Placement > Parameter Count: The stability of fine-tuning depends more on where you put adapters than how many you use. For instance, the planner explicitly excludes depthwise convolutions and fixed DFL projections because small updates to these can corrupt the loss geometry.
  • The Memory-Time Trade-off: While PEFT is famous for saving memory, it isn't a "speedup" for training. The authors found that while LoRA reduced peak VRAM by 43.9%, the overhead of managing sidecar adapters actually made training 1.72x longer than full fine-tuning.
  • Architecture-Conditioned Reliability: PEFT reliability is tied to the architecture. Standard LoRA variants that work perfectly on CNN-only YOLO11n failed catastrophically on the attention-heavy YOLO12n and RT-DETR-L. This underscores the need for the framework's "Refuse" mechanism.

Key Results

The results demonstrate that a well-planned PEFT strategy can actually beat full fine-tuning (Full-SFT):

  • Accuracy: On YOLO11s and YOLO12s, planner-selected RS-LoRA reached 0.7138 and 0.7307 mAP50-95, respectively. This is a massive jump compared to 0.6428 and 0.6662 for Full-SFT.
  • Safety: For the Transformer-based RT-DETR-L, the planner correctly identified that all LoRA configurations would fail and returned a "Refuse" decision, directing the user to use Full-SFT instead.
  • Efficiency: The framework successfully reduced peak VRAM from 28.57 GB to 16.03 GB on the audited YOLO11 configuration.
Table 1. Performance matrix comparing various PEFT variants against Full-SFT anchors.

Limitations and Future Work

The authors acknowledge that while the framework is robust for the evaluated families, refusal on completely unseen detector architectures remains an open validation problem. The current reliability calibration is based on observed failures within known families.

Future work aims to extend these rules to Vision-Language Model (VLM) detectors with cross-modal attention and to more complex Mixture-of-Transformers (MoT) architectures. The team also plans to explore more sophisticated budget-aware dispatching for MoE detectors.

Project Page: github.com/Tencent/YOLO-Master

Summary updated

Abstract

Generic parameter-efficient fine-tuning (PEFT) methods transferred from language models can fail silently on real-time detectors, whose heterogeneous operators and detection-specific components impose placement constraints absent from regular Transformer stacks. We propose YOLO-PEFT, a structure-aware framework that formulates adapter placement as an auditable constraint-planning problem. Given a detector graph, a PEFT request, and a resource budget, YOLO-PEFT assigns operator and semantic roles, evaluates explicit operator-validity, detector-semantic, graph-interface, and deployment predicates, records a reason code for each excluded module, and either emits a budgeted target-module plan or returns Refuse before training. Under the official VOC07+12 trainval-to-VOC07 test protocol, planner-selected RS-LoRA reaches 0.7138 and 0.7307 mAP50-95 on YOLO11s and YOLO12s, respectively, compared with 0.6428 and 0.6662 for Full-SFT. On RT-DETR-L, all seven evaluated LoRA-family configurations cross the predefined catastrophic threshold, supporting a calibrated Refuse-to-Full-SFT decision within the evaluated coverage. A controlled YOLO11 audit further shows that LoRA reduces peak training memory by 43.9 percent, although training takes 1.72 times longer. Within the evaluated detector families, placement policies, and calibration coverage, YOLO-PEFT replaces manual target-module trial and error with explicit, inspectable planning while preserving verified train-save-merge-export paths; refusal on unseen detector architectures remains an open validation problem. Project Page: github.com/Tencent/YOLO-Master
BibTeX
@misc{lin2026yolo,
  author = {Xu Lin and WenJie Nie and Jinlong Peng and Weifu Fu and YueXiao Ma and Xiawu Zheng and Yong Liu},
  title = {{YOLO-PEFT: Parameter-Efficient Fine-Tuning on YOLO Family}},
  year = {2026},
  eprint = {2608.07051},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.07051},
}
Figure