Abstract
Recent advances in parameter-efficient transfer learning have demonstrated the utility of composing LoRA adapters from libraries of pretrained modules. However, most existing approaches rely on simple retrieval heuristics or uniform averaging, which overlook the latent structure of task relationships in representation space. We propose a new framework for adapter reuse that moves beyond retrieval, formulating adapter composition as a geometry-aware sparse reconstruction problem. Specifically, we represent each task by a latent prototype vector derived from the base model's encoder and aim to approximate the target task prototype as a sparse linear combination of retrieved reference prototypes, under an $\ell_1$-regularized optimization objective. The resulting combination weights are then used to blend the corresponding LoRA adapters, yielding a composite adapter tailored to the target task. This formulation not only preserves the local geometric structure of the task representation manifold, but also promotes interpretability and efficient reuse by selecting a minimal set of relevant adapters. We demonstrate the effectiveness of our approach across multiple domains—including medical image segmentation, medical report generation and image synthesis. Our results highlight the benefit of coupling retrieval with latent geometry-aware optimization for improved zero-shot generalization.
Introduction
Foundation models such as Llama, SAM, and Stable Diffusion are powerful but computationally expensive to fine-tune for every specific niche. Low-Rank Adaptation (LoRA) has mitigated this by creating small, modular "adapters" for specific tasks. However, as adapter libraries grow into the hundreds of thousands, a new challenge emerges: how to effectively reuse them for tasks the model hasn't seen before.
Existing methods typically employ simple retrieval (picking the single most similar adapter) or uniform fusion (averaging several together). These approaches often fail because they ignore the "latent geometry"—the underlying structural relationships—between different tasks. Simple averaging can lead to parameter interference, while picking just one adapter might miss complementary knowledge from others.
Methodology: Latent Geometry-Preserving Composition
We propose a three-stage pipeline called Latent Geometry-Preserving Composition:
- Task Prototype Construction: Every task in the library is represented as a "prototype vector" ($z_i$), created by averaging the feature representations of its data through the base model’s encoder.
- Sparse Task Projection: When a new target task arrives, the system retrieves the $k$ most similar reference prototypes. Instead of simple averaging, it solves an $\ell_1$-regularized optimization problem to find a sparse linear combination of these reference prototypes that best reconstructs the target task's prototype.
- Adapter Reassembly: The weights ($w_i$) found in the previous step—which can be positive or negative—are used to linearly combine the actual LoRA parameters ($\delta\theta$) into a new composite adapter.
This approach is "geometry-aware" because it assumes that the relationships between tasks in the embedding space should be preserved when blending their parameters.
Key Insights
- Sparsity as Interpretability: By using $\ell_1$ regularization, the model naturally selects only a few highly relevant adapters. This makes the composition interpretable, allowing users to see exactly which source tasks contribute to a new task.
- The Power of Linear Combinations: Unlike similarity-based weighting, which only uses positive weights, this method allows for negative weights. This helps the model "subtract" irrelevant features or noise, leading to sharper results.
Experimental Results
The researchers validated their method across three distinct domains:
- Medical Image Segmentation: Using the Segment Anything Model (SAM), the sparse projection method consistently outperformed uniform averaging. For example, on the "AbdAtlas" dataset, the DICE score improved from 19.2% (Average) to 64.5%.
- Medical Report Generation: Using Llama 3.1 8B, the method improved zero-shot performance on radiology report datasets.
- Image Synthesis: In image generation with Stable Diffusion v1.5, both human raters and GPT-4o preferred images generated by this method over baselines, noting better adherence to complex prompts and higher visual quality.
The entire process happens at inference time, requiring no gradient-based optimization or new labeled data. Composing an adapter takes less than three minutes per task, representing a 10x speedup over traditional fine-tuning.
Conclusion
This framework moves beyond simple adapter retrieval by treating composition as a geometry-aware reconstruction problem. By preserving the latent structure of task relationships, it enables effective zero-shot generalization and efficient reuse of large-scale adapter libraries. Future work includes scaling to million-scale libraries and exploring multi-modal adapter composition.
Summary updated