PLEAT: Progressive LoRA Expert Aggregation and Transport for Continual Model Merging

Jianchun Yang, Jian Liang, Xianda Guo, Yanlun Peng, Pinhan Fu, Ke He, Wenke Huang, Mang YeOpenReview 2027 · Submission to ICLR 2027 · Withdrawn · OpenReview

Its authors withdrew this paper, so there is no full text to read or summarize.

Abstract

Continual learning aims to acquire knowledge from new tasks while retaining previously learned capabilities. One way to pursue this goal is through model merging, which combines the weights of independently trained experts without additional training. A central challenge in continual model merging is to integrate new experts without retaining an expanding expert pool. In resource-constrained settings, merging experts trained with low-rank adaptation (LoRA) must also preserve their compact representation. Existing merging strategies face three limitations in this setting: (1) joint merging requires retaining and revisiting an expanding collection of historical experts; (2) methods designed for fully fine-tuned models typically operate on dense updates, giving up LoRA’s compact representation when applied directly to LoRA experts; and (3) fixed-rank LoRA merging preserves compactness but can become a capacity bottleneck as knowledge accumulates over long task sequences. We propose PLEAT, Progressive LoRA Expert Aggregation and Transport, a training-free approach that exploits the low-rank structure of LoRA experts to maintain a factorized merged update with progressively adjustable capacity. PLEAT incrementally updates a shared input basis under a prescribed rank schedule, aligns each incoming expert to the updated representation, and transports historical coefficients before aggregation. Experiments across multiple backbone models show that PLEAT outperforms a range of existing merging methods, while retaining the parameter-efficient representation of LoRA and enabling continual merging without revisiting historical expert weights.
BibTeX
@misc{yang2027pleat,
  author = {Jianchun Yang and Jian Liang and Xianda Guo and Yanlun Peng and Pinhan Fu and Ke He and Wenke Huang and Mang Ye},
  title = {{PLEAT: Progressive LoRA Expert Aggregation and Transport for Continual Model Merging}},
  year = {2027},
  url = {https://openreview.net/forum?id=urlGJH4Aq8},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2027},
}

Loading related content...

Figure