RewardDS: Privacy-Preserving Fine-Tuning for Large Language Models via Reward Driven Data Synthesis

J.h. Wang, C.‐j. Richard Shi, Jingjing Yang, Haoran Li, Qianli Ma, Huiping Zhuang, Cen Chen, Ziqian Zeng2025 · DOI

Introduction

The rapid advancement of Large Language Models (LLMs) has created a significant demand for domain-specific fine-tuning. Organizations in sectors such as healthcare and finance possess vast quantities of sensitive data that could significantly enhance LLM performance. However, the privacy risks associated with uploading this data to third-party cloud servers are substantial. Standard fine-tuning practices often risk the memorization and subsequent leakage of sensitive information.

To mitigate these risks, researchers have turned to synthesis-based fine-tuning. In this paradigm, a client locally trains a small "proxy" model on private data using Differential Privacy (DP) and then uses this proxy to generate synthetic training data on the server. While this protects the original data, the resulting synthetic datasets are often plagued by noise, incoherence, and factual inaccuracies, leading to suboptimal performance in the target LLM. This paper introduces RewardDS, a framework designed to bridge the gap between privacy and data quality through reward-driven data synthesis.

The RewardDS Framework

RewardDS addresses the quality bottleneck of synthetic data by introducing a dual-model approach on the client side and a refinement loop on the server side. The framework operates across a client-server architecture with three primary stages:

1. Client-Side Private Training

Instead of training only a generation model, the client trains two lightweight models: a Generation Proxy Model and a Reward Proxy Model. Both models are trained using Differentially Private Stochastic Gradient Descent (DP-SGD), which provides a mathematical guarantee that individual data points cannot be extracted from the models. The Reward Proxy is specifically trained on paired comparison data to learn the nuances of domain-specific quality.

2. Server-Side Generation and Filtering

Once the DP-trained proxies are sent to the server, the Generation Proxy produces raw synthetic queries and responses. The Reward Proxy then evaluates these samples. Unlike traditional methods that rely on surface-level text similarity (like ROUGE or embedding distance), the Reward Proxy identifies whether a response is actually useful and accurate for the specific domain. Low-scoring, noisy samples are filtered out to prevent them from degrading the target LLM's performance.

3. Self-Optimizing Refinement

Filtering alone can significantly reduce the size of the training set, potentially leading to overfitting. RewardDS overcomes this through an iterative refinement process. The system generates multiple candidate responses for a single query; the Reward Proxy identifies the best and worst candidates, and the Generation Proxy provides feedback explaining the differences. The target LLM then uses this feedback to synthesize a high-quality, refined response, maintaining a robust dataset size while maximizing quality.

Experimental Evaluation

The authors evaluated RewardDS across three diverse, high-stakes domains using public datasets treated as private client data:

  • Medical QA: HealthCareMagic-100k
  • Financial QA: fingpt-fiqa_qa
  • Code Generation: opc-sft-stage2

Performance Results

RewardDS demonstrated significant performance gains over existing privacy-preserving baselines. In the Medical QA task, RewardDS achieved a ROUGE-1 score of 27.78, a dramatic improvement over standard DP-Generation (16.22) and similarity-based filtering (11.94). In Code Generation, the framework reached a Pass@1 score of 32.41, compared to 25.51 for the baseline.

Efficiency and Scalability

Despite the additional steps of reward scoring and iterative refinement, RewardDS remains computationally practical. The authors reported that the framework only added approximately 30% to the total server-side training time, making it a viable solution for real-world deployment where time and compute resources are constrained.

Privacy and Security Analysis

A key insight of the paper is that splitting the privacy budget between a generator and a reward model actually enhances security. By allocating a portion of the total privacy budget (epsilon) to each model, more noise is injected into the training process of each individual component. Empirical testing via Data Extraction and Membership Inference attacks showed that RewardDS is significantly less vulnerable to privacy breaches than single-model baselines.

Conclusion and Future Work

RewardDS represents a significant step forward in privacy-preserving LLM fine-tuning. By shifting the focus from text similarity to reward-driven quality, the framework allows organizations to leverage sensitive data without compromising user privacy. While current experiments utilized LoRA (Low-Rank Adaptation) due to hardware constraints, future work will explore full-parameter fine-tuning and the scalability of RewardDS on even larger LLM backbones. Additionally, the authors aim to further optimize client-side efficiency to reduce the computational burden on data owners.

Summary updated

Abstract

The success of large language models (LLMs) has attracted many individuals to fine-tune them for domain-specific tasks by uploading their data.However, in sensitive areas like healthcare and finance, privacy concerns often arise.One promising solution is to generate synthetic data with Differential Privacy (DP) guarantees to replace private data.However, these synthetic data contain significant flawed data, which are considered as noise.Existing solutions typically rely on naive filtering by comparing ROUGE-L scores or embedding similarities, which are ineffective in addressing the noise.To address this issue, we propose RewardDS, a novel privacy-preserving framework that fine-tunes a reward proxy model and uses reward signals to guide the synthetic data generation.Our RewardDS introduces two key modules, Reward Guided Filtering and Self-Optimizing Refinement, to both filter and refine the synthetic data, effectively mitigating the noise.Extensive experiments across medical, financial, and code generation domains demonstrate the effectiveness of our method.Our code and
BibTeX
@misc{wang2025rewardds,
  author = {J.H. Wang and C.‐J. Richard Shi and Jingjing Yang and Haoran Li and Qianli Ma and Huiping Zhuang and Cen Chen and Ziqian Zeng},
  title = {{RewardDS: Privacy-Preserving Fine-Tuning for Large Language Models via Reward Driven Data Synthesis}},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.223},
}
Figure