TL;DR
ResMerge is a novel, training-free model merging framework designed specifically for reinforcement learning (RL) expert LLMs. It challenges the conventional wisdom that a task's signal is purely concentrated in the top singular values of its weight updates, revealing that RL experts store substantial, stable behavioral knowledge in their spectral residuals. By building a stable backbone from these residuals and selectively injecting top spectral components via an agreement-aware gate, ResMerge successfully blends diverse expert capabilities while minimizing task interference.
What Problem Does This Paper Address?
Model merging allows researchers to combine multiple specialized "expert" language models (e.g., a math expert, a coding expert, and a reasoning expert) into a single multitasking model without the massive computational cost of retraining. This is typically done by manipulating task vectors—the parameter differences between a fine-tuned expert and the original base model.
However, directly adding or averaging these vectors often causes destructive interference, where conflicting weight updates cancel each other out. To fix this, recent methods have turned to Singular Value Decomposition (SVD) to isolate and preserve the high-energy "leading spectral directions" (the heads) while compressing or discarding the lower-energy "residuals" as noise.
The authors discover that this assumption completely breaks down for models trained via Reinforcement Learning (RL). In RL experts, the spectral residual is not noise; it contains vital behavioral knowledge. Discarding it severely degrades performance, yet directly averaging the high-energy heads causes sharp cross-expert conflicts.
Methods
ResMerge addresses this functional heterogeneity by explicitly splitting each layer's task matrix into a rank-$k$ spectral head ($H$) and a residual component ($R$). It then applies a two-step component-wise merging strategy:
- Spherical Residual Consensus Adaptation (SRC-A): Because residual components are spectrally dispersed and geometrically stable, they form the merging backbone. Instead of plain Euclidean averaging (which dilutes weak shared signals), SRC-A projects the residuals onto a Frobenius unit sphere to find a consensus direction. The magnitude of this backbone is dynamically scaled based on a reliability score derived from cross-expert consensus alignment.
- Lightweight Head Correction (LHC): The highly concentrated spectral heads are informative but risky to merge directly. ResMerge treats the average head as a candidate correction vector. It injects this correction into the residual backbone only if it passes through a "positive agreement gate" ($S_H$), which measures directional alignment across the experts' heads. If the experts sharply disagree, the head correction is suppressed to prevent interference.
Datasets
The authors evaluated ResMerge across multiple model families (Qwen2.5-7B-Base, Qwen3-4B-Base, Qwen2.5-7B-Instruct, and Llama-3.2-3B-Instruct) spanning five major capability domains:
- Coding: LiveCodeBench (release_v2), HumanEvalPlus, and MBPPPlus.
- General Reasoning: GPQA-Diamond.
- Mathematics: AIME24, AIME25, AMC23, and MATH500.
- Tool Use: Berkeley Function Calling Leaderboard (BFCL V3).
- Memory: HotpotQA and SQuAD (sampled subsets).
No new benchmarks were introduced; the authors relied entirely on these standard, rigorous evaluation suites.
Key Insights and Ideas
- The RL Residual Surprise: In standard Supervised Fine-Tuning (SFT), removing the top spectral head destroys model performance. In RL experts, however, a model consisting only of the residual weights can match or even exceed the performance of the full task vector (see Figure 1).
- Spectral Concentration: RL training alters weights in a less uniform way than SFT, packing massive energy into a few dominant directions. This concentration creates localized, severe merging risks if treated homogeneously.
- Backbone vs. Correction: Because residuals exhibit much higher spherical coherence across different experts than heads do, they serve as a much safer foundation for a unified model. The heads are better utilized as cautious, gated "add-ons."
Key Results
- State-of-the-Art Merging: On the Qwen2.5-7B-Base expert group, ResMerge achieved an overall average score of 41.08%, outperforming the strongest non-ours spectral baseline (ISO-C) by 2.22 percentage points and standard Task Arithmetic by 2.42 points.
- Balanced Preservation: Rather than excelling at one task while tanking others, ResMerge demonstrated superior expert-capability retention, significantly reducing the Worst Capability Shortfall (WCS) compared to baselines.
- Efficiency: ResMerge completed its offline merge in 2.22 minutes on 8× NVIDIA A6000 GPUs—faster than competing spectral methods like TSV-Merge (3.65 mins) and ISO-CTS (3.23 mins)—while maintaining a low peak GPU memory footprint (8.13 GB) equivalent to SVD-free methods.
Limitations and Future Work
- Static Parameter Merging: ResMerge operates purely in the parameter space without input-dependent routing (like Mixture-of-Experts).
- Coarse Head Composition: The current LHC module uses a simple averaged rank-1 head for correction. The authors note that a more fine-grained or task-aware head composition could yield better results when experts have highly diverse behaviors.
- Safety Risks: As a parameter-space merger, the final model can inherit latent biases or safety risks present in the source experts, necessitating downstream safety evaluations before deployment.
Summary updated