ResMerge: Residual-based Spectral Merging of Large Language Models

Yandu Sun, Zhiyan Hou, Hongyan An, Weizhen Wang, Haokai Ma, Yuheng Jia, Junfeng Fang, Haiyun Guo, Jinqiao WangarXiv 2026 · arXiv

TL;DR

ResMerge is a novel, training-free model merging framework designed specifically for reinforcement learning (RL) expert LLMs. It challenges the conventional wisdom that a task's signal is purely concentrated in the top singular values of its weight updates, revealing that RL experts store substantial, stable behavioral knowledge in their spectral residuals. By building a stable backbone from these residuals and selectively injecting top spectral components via an agreement-aware gate, ResMerge successfully blends diverse expert capabilities while minimizing task interference.

What Problem Does This Paper Address?

Model merging allows researchers to combine multiple specialized "expert" language models (e.g., a math expert, a coding expert, and a reasoning expert) into a single multitasking model without the massive computational cost of retraining. This is typically done by manipulating task vectors—the parameter differences between a fine-tuned expert and the original base model.

However, directly adding or averaging these vectors often causes destructive interference, where conflicting weight updates cancel each other out. To fix this, recent methods have turned to Singular Value Decomposition (SVD) to isolate and preserve the high-energy "leading spectral directions" (the heads) while compressing or discarding the lower-energy "residuals" as noise.

The authors discover that this assumption completely breaks down for models trained via Reinforcement Learning (RL). In RL experts, the spectral residual is not noise; it contains vital behavioral knowledge. Discarding it severely degrades performance, yet directly averaging the high-energy heads causes sharp cross-expert conflicts.

Component-level recovery under RL and SFT post-training.

Methods

ResMerge addresses this functional heterogeneity by explicitly splitting each layer's task matrix into a rank-$k$ spectral head ($H$) and a residual component ($R$). It then applies a two-step component-wise merging strategy:

  1. Spherical Residual Consensus Adaptation (SRC-A): Because residual components are spectrally dispersed and geometrically stable, they form the merging backbone. Instead of plain Euclidean averaging (which dilutes weak shared signals), SRC-A projects the residuals onto a Frobenius unit sphere to find a consensus direction. The magnitude of this backbone is dynamically scaled based on a reliability score derived from cross-expert consensus alignment.
  2. Lightweight Head Correction (LHC): The highly concentrated spectral heads are informative but risky to merge directly. ResMerge treats the average head as a candidate correction vector. It injects this correction into the residual backbone only if it passes through a "positive agreement gate" ($S_H$), which measures directional alignment across the experts' heads. If the experts sharply disagree, the head correction is suppressed to prevent interference.
Overview of the ResMerge framework.

Datasets

The authors evaluated ResMerge across multiple model families (Qwen2.5-7B-Base, Qwen3-4B-Base, Qwen2.5-7B-Instruct, and Llama-3.2-3B-Instruct) spanning five major capability domains:

  • Coding: LiveCodeBench (release_v2), HumanEvalPlus, and MBPPPlus.
  • General Reasoning: GPQA-Diamond.
  • Mathematics: AIME24, AIME25, AMC23, and MATH500.
  • Tool Use: Berkeley Function Calling Leaderboard (BFCL V3).
  • Memory: HotpotQA and SQuAD (sampled subsets).

No new benchmarks were introduced; the authors relied entirely on these standard, rigorous evaluation suites.

Key Insights and Ideas

  • The RL Residual Surprise: In standard Supervised Fine-Tuning (SFT), removing the top spectral head destroys model performance. In RL experts, however, a model consisting only of the residual weights can match or even exceed the performance of the full task vector (see Figure 1).
  • Spectral Concentration: RL training alters weights in a less uniform way than SFT, packing massive energy into a few dominant directions. This concentration creates localized, severe merging risks if treated homogeneously.
  • Backbone vs. Correction: Because residuals exhibit much higher spherical coherence across different experts than heads do, they serve as a much safer foundation for a unified model. The heads are better utilized as cautious, gated "add-ons."

Key Results

  • State-of-the-Art Merging: On the Qwen2.5-7B-Base expert group, ResMerge achieved an overall average score of 41.08%, outperforming the strongest non-ours spectral baseline (ISO-C) by 2.22 percentage points and standard Task Arithmetic by 2.42 points.
  • Balanced Preservation: Rather than excelling at one task while tanking others, ResMerge demonstrated superior expert-capability retention, significantly reducing the Worst Capability Shortfall (WCS) compared to baselines.
  • Efficiency: ResMerge completed its offline merge in 2.22 minutes on 8× NVIDIA A6000 GPUs—faster than competing spectral methods like TSV-Merge (3.65 mins) and ISO-CTS (3.23 mins)—while maintaining a low peak GPU memory footprint (8.13 GB) equivalent to SVD-free methods.

Limitations and Future Work

  • Static Parameter Merging: ResMerge operates purely in the parameter space without input-dependent routing (like Mixture-of-Experts).
  • Coarse Head Composition: The current LHC module uses a simple averaged rank-1 head for correction. The authors note that a more fine-grained or task-aware head composition could yield better results when experts have highly diverse behaviors.
  • Safety Risks: As a parameter-space merger, the final model can inherit latent biases or safety risks present in the source experts, necessitating downstream safety evaluations before deployment.

Summary updated

Abstract

Model merging offers a training-free way to combine multiple post-trained expert models, but merging experts obtained through reinforcement learning (RL) remains challenging. Existing spectral merging methods often assume that leading singular directions contain the main task signal, while lower-energy residual components can be compressed, selected, or attenuated to reduce interference. We find that this assumption does not hold for RL task vectors: after decomposing each task vector into a leading spectral head and a residual component, both parts can independently recover substantial behavior knowledge, while exhibiting different merging properties. The head is highly concentrated and informative but more prone to sharp cross-expert conflicts, whereas the residual component is more dispersed and provides a more stable basis for aggregation. Based on this observation, we propose ResMerge, a residual-based spectral merging framework for RL experts. ResMerge first constructs a stable residual backbone with Spherical Residual Consensus Adaptation, which estimates a reliability-weighted consensus direction on the Frobenius sphere. It then reintroduces leading-head information through a Lightweight Head Correction module gated by positive cross-expert agreement. Experiments across multiple RL expert groups and capability domains show that ResMerge better preserves expert capabilities than representative task-vector and spectral merging baselines.
BibTeX
@misc{sun2026resmerge,
  author = {Yandu Sun and Zhiyan Hou and Hongyan An and Weizhen Wang and Haokai Ma and Yuheng Jia and Junfeng Fang and Haiyun Guo and Jinqiao Wang},
  title = {{ResMerge: Residual-based Spectral Merging of Large Language Models}},
  year = {2026},
  eprint = {2606.02252},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.02252},
}

Loading related content...

Figure