FedProxy: Federated Fine-Tuning of LLMs via Proxy SLMs and Heterogeneity-Aware Fusion

Tao Fan, Guoqiang Ma, Yuanfeng Song, Lixin Fan, Kai Chen, Qiang Yang

TL;DR

FedProxy is a new framework for federated fine-tuning of Large Language Models (LLMs) that solves the "trilemma" of protecting model intellectual property, ensuring client privacy, and maintaining high performance on diverse data. By using a compressed "Proxy Small Language Model" (SLM) as a high-fidelity surrogate for client-side training and a conflict-aware fusion mechanism, it achieves up to 90% of centralized fine-tuning performance—significantly outperforming existing adapter-based methods.

What Problem Does This Paper Address?

Fine-tuning LLMs on decentralized, private data (like medical records or proprietary code) is a major challenge. Researchers face a "trilemma":

  1. IP Protection: Model owners don't want to distribute their full, multi-billion parameter "secret sauce" to clients.
  2. Data Privacy: Clients cannot share their sensitive data with the model owner.
  3. Data Heterogeneity: Different clients have different data distributions (non-IID), which often causes "negative interference" when merging their updates, tanking the final model's performance.

Existing solutions like Offsite-Tuning (OT) try to solve this by sending clients a lightweight "emulator" and trainable adapters. However, these adapters often lack the representational capacity to capture complex knowledge, leaving a massive performance gap compared to training the full model centrally.

Methods

FedProxy introduces a three-stage architecture to bridge this gap:

  1. Server-Guided Compression: Instead of weak adapters, the server creates a Proxy SLM. It uses public data (like Alpaca) to identify the most influential transformer blocks in the original LLM using a Block Influence (BI) metric. It then prunes the less important blocks to create a compact, high-fidelity sub-network that clients can actually afford to train.
  2. Federated Aggregation with Interference Mitigation: To handle messy, heterogeneous data, FedProxy uses two novel strategies:
  3. PCR (Parameter-Level Conflict-Aware Regularization): The server tracks which parameters clients disagree on. Clients then use these "conflict scores" to regularize their local training, staying close to the global model where there is consensus but allowing flexibility where tasks differ.
  4. H-TIES (Heterogeneity-aware TIES-merging): A sophisticated merging algorithm that sparsifies updates from "outlier" clients and resolves sign conflicts (e.g., if one client wants to increase a weight and another wants to decrease it) to prevent updates from canceling each other out.
  5. Training-Free "Plug-in" Fusion: Once the Proxy SLM is fine-tuned, its weights are directly "plugged back" into the original LLM. Because the SLM is a direct sub-network of the LLM, this is a simple parameter replacement that requires zero extra training and adds zero inference latency.
Overview of the FedProxy framework

Datasets

The authors evaluated FedProxy using two foundational models: LLaMA2-7B and Mistral-7B-Instruct-v0.2.

  • Compression Data: The public Alpaca dataset was used for the initial distillation/pruning phase.
  • Evaluation Benchmarks:
    • Question Answering (QA): OBQA, ARC-Challenge, ARC-Easy, and CommonsenseQA.
  • General Language Understanding (GLUE): SST2, MRPC, RTE, and MNLI.
  • Scenarios: They tested both Homogeneous (IID data across 4 clients) and Heterogeneous (unique datasets for each of 8 clients) environments.

Key Insights and Ideas

  • SLMs as Semantic Vessels: A key "aha" moment is that a compressed version of the full model architecture is a much better learner than a disjointed adapter-emulator setup. It retains the structural knowledge of the backbone while remaining computationally feasible for edge devices.
  • Conflict-Awareness is Crucial: Parameter interference isn't just a global problem; it happens at the level of individual dimensions. By providing clients with conflict scores during training (PCR), the framework prevents interference before it even reaches the server for merging.
  • Pragmatic IP Protection: While not cryptographically secure, the initial Proxy SLM is significantly weaker than the full LLM (as shown in their IP analysis), making it a poor target for theft while still serving as an excellent training surrogate.

Key Results

FedProxy consistently outperformed all federated baselines (like FedOT and FedBiOT) and approached the "gold standard" of centralized training (CentSFT).

  • Performance Gains: For LLaMA2-7B at 50% compression, FedProxy improved QA accuracy by 11.7% and GLUE accuracy by 22.1% over the standard FedOT baseline.
  • Closing the Gap: It reached 90.1% of centralized performance on QA and 90.7% on GLUE tasks.
  • Efficiency: While it requires more iterative communication than simple adapter methods, it maintains a comparable initial download size and delivers much higher utility.
Method Performance Comparison in the Homogeneous Tasks Setting

Limitations and Future Work

  • Compression Floor: Performance drops significantly if you prune too much. A 50% compression ratio is the "sweet spot"; at 70%, the proxy loses too much representational capacity.
  • Computational Scalability: The current conflict-aware aggregation scales quadratically ($O(K^2)$) with the number of clients, which might be a bottleneck for massive-scale deployments.
  • Catastrophic Forgetting: The authors note that the long-term stability of the global backbone after many rounds of "plug-in" fusion needs further study to ensure the model doesn't lose its general-purpose capabilities.

Summary updated

Abstract

Federated fine-tuning of Large Language Models (LLMs) is obstructed by a trilemma of challenges: protecting LLMs intellectual property (IP), ensuring client privacy, and mitigating performance loss on heterogeneous data. Existing methods like Offsite-Tuning (OT) secure the LLMs IP by having clients train only lightweight adapters, yet our analysis reveals they suffer from a fundamental performance bottleneck, leaving a significant gap compared to centralized training. To bridge this gap, we introduce FedProxy, a new federated adaptation framework. FedProxy replaces weak adapters with a unified, powerful Proxy Small Language Model (SLM), compressed from the proprietary LLM, to serve as a high-fidelity surrogate for collaborative fine-tuning. Our framework systematically resolves the trilemma through a three-stage architecture: (i) Efficient Representation via server-guided compression to create a resource-friendly proxy; (ii) Robust Optimization through an interference-mitigating aggregation strategy to handle data heterogeneity; and (iii) Effortless Fusion via a training-free "plug-in" mechanism to integrate learned knowledge back into the LLM. Experiments show FedProxy significantly outperforms OT methods and approaches centralized performance, establishing a new benchmark for secure and high-performance federated LLM adaptation.
BibTeX
@inproceedings{fan2026fedproxy,
  author = {Tao Fan and Guoqiang Ma and Yuanfeng Song and Lixin Fan and Kai Chen and Qiang Yang},
  title = {{FedProxy: Federated Fine-Tuning of LLMs via Proxy SLMs and Heterogeneity-Aware Fusion}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.794},
}

Loading related content...

Figure