Abstract
From a multi-model compression perspective, model merging enables memory-efficient serving of multiple models fine-tuned from the same base, but suffers from degraded performance due to interference among their task-specific parameter adjustments (i.e., deltas). In this paper, we reformulate model merging as a compress-and-retrieve scheme, revealing that the task interference arises from the summation of irrelevant deltas during model retrieval. To address this issue, we use random orthogonal transformations to decorrelate these vectors into self-cancellation. We show that this approach drastically reduces interference, improving performance across both vision and language tasks. Since these transformations are fully defined by random seeds, adding new models requires no extra memory. Further, their data- and model-agnostic nature enables easy addition or removal of models with minimal compute overhead, supporting efficient and flexible multi-model serving.
Introduction
In the current landscape of foundation models, researchers and practitioners often manage a "zoo" of models: a single base model (such as CLIP or Llama) fine-tuned into dozens of specialized versions for distinct tasks. Storing and serving these models individually presents a significant memory and storage challenge.
While the community has explored model merging—averaging the weights of these models—this approach typically incurs a performance "tax." As more models are merged, their task-specific adjustments, or deltas, begin to interfere with one another. This interference acts like static in a radio signal; as more tasks are added, the signal-to-noise ratio drops until the original task performance is lost. RanDeS (Randomized Delta Superposition) introduces a framework to "superpose" these models so they can be retrieved cleanly without this interference.
Methods: Randomized Delta Superposition
The core innovation of RanDeS is the application of random orthogonal transformations to each model's delta before merging them into a shared buffer.
Because modern neural networks operate in extremely high-dimensional spaces, random transformations are highly likely to be orthogonal (perpendicular) to one another. When a specific model (e.g., "Model 1") needs to be retrieved, the inverse transformation is applied to the superposed buffer. This process preserves the signal for the target model while causing the deltas from all other models to decorrelate and effectively cancel each other out—a phenomenon termed self-cancellation.
RanDeS offers three lightweight, training-free implementations:
- RanDeS-S (Layer Shuffling): Randomly permuting layers of the same type within a model.
- RanDeS-RSF (Random Sign Flips): Randomly flipping the signs of columns in the weight matrices.
- RanDeS-SRSF: A combination of both for maximum decorrelation.
Crucially, these transformations are defined entirely by random seeds. Adding a new model to the compressed pool requires virtually zero extra memory—only the storage of a new seed.
Key Insights
- Interference as Summed Noise: Task interference in merged models is mathematically equivalent to the summation of irrelevant deltas. By ensuring these deltas are orthogonal, the interference drops toward zero.
- The Blessing of Dimensionality: Simple random shuffling or sign-flipping is sufficient to induce orthogonality in high-dimensional weight spaces, removing the need for complex optimization.
- Dynamic Model Management: Unlike methods that require re-calculating global parameters or "routers" when adding tasks, RanDeS allows for the independent addition or removal of models. This "hot-swapping" capability is enabled by the data-agnostic nature of the random transformations.
Experimental Results
The authors evaluated RanDeS across diverse vision and language benchmarks, including CLIP-ViT, Flan-T5, and GPT-2, across dozens of classification and generation tasks.
As shown in the results table, RanDeS-SRSF achieves performance nearly identical to individual fine-tuned models. On the CLIP-ViT-B/32 benchmark, it reached 89.9% average accuracy, compared to 90.3% for separate models, significantly outperforming standard task arithmetic (69.8%) and other merging baselines. Furthermore, the method demonstrated extreme scalability, successfully merging 20 different CLIP-ViT-L/14 models into a single footprint while maintaining 93.5% accuracy (vs. 94.2% for individual models).
Conclusion and Future Work
RanDeS provides a highly efficient path for multi-model serving, reducing storage costs by 4x to 9x without the typical performance degradation of model merging. While the current implementation requires an initial memory overhead for the base weights and a delta buffer, it excels at scaling to numerous tasks. Future research may explore fine-tuning strategies that explicitly encourage delta orthogonality or the use of learned systematic transformations to further close the gap with individual model performance.
Summary updated