Introduction
The rapid advancement of Large Language Models (LLMs) has created a significant demand for domain-specific fine-tuning. Organizations in sectors such as healthcare and finance possess vast quantities of sensitive data that could significantly enhance LLM performance. However, the privacy risks associated with uploading this data to third-party cloud servers are substantial. Standard fine-tuning practices often risk the memorization and subsequent leakage of sensitive information.
To mitigate these risks, researchers have turned to synthesis-based fine-tuning. In this paradigm, a client locally trains a small "proxy" model on private data using Differential Privacy (DP) and then uses this proxy to generate synthetic training data on the server. While this protects the original data, the resulting synthetic datasets are often plagued by noise, incoherence, and factual inaccuracies, leading to suboptimal performance in the target LLM. This paper introduces RewardDS, a framework designed to bridge the gap between privacy and data quality through reward-driven data synthesis.
The RewardDS Framework
RewardDS addresses the quality bottleneck of synthetic data by introducing a dual-model approach on the client side and a refinement loop on the server side. The framework operates across a client-server architecture with three primary stages:
1. Client-Side Private Training
Instead of training only a generation model, the client trains two lightweight models: a Generation Proxy Model and a Reward Proxy Model. Both models are trained using Differentially Private Stochastic Gradient Descent (DP-SGD), which provides a mathematical guarantee that individual data points cannot be extracted from the models. The Reward Proxy is specifically trained on paired comparison data to learn the nuances of domain-specific quality.
2. Server-Side Generation and Filtering
Once the DP-trained proxies are sent to the server, the Generation Proxy produces raw synthetic queries and responses. The Reward Proxy then evaluates these samples. Unlike traditional methods that rely on surface-level text similarity (like ROUGE or embedding distance), the Reward Proxy identifies whether a response is actually useful and accurate for the specific domain. Low-scoring, noisy samples are filtered out to prevent them from degrading the target LLM's performance.
3. Self-Optimizing Refinement
Filtering alone can significantly reduce the size of the training set, potentially leading to overfitting. RewardDS overcomes this through an iterative refinement process. The system generates multiple candidate responses for a single query; the Reward Proxy identifies the best and worst candidates, and the Generation Proxy provides feedback explaining the differences. The target LLM then uses this feedback to synthesize a high-quality, refined response, maintaining a robust dataset size while maximizing quality.
Experimental Evaluation
The authors evaluated RewardDS across three diverse, high-stakes domains using public datasets treated as private client data:
- Medical QA: HealthCareMagic-100k
- Financial QA: fingpt-fiqa_qa
- Code Generation: opc-sft-stage2
Performance Results
RewardDS demonstrated significant performance gains over existing privacy-preserving baselines. In the Medical QA task, RewardDS achieved a ROUGE-1 score of 27.78, a dramatic improvement over standard DP-Generation (16.22) and similarity-based filtering (11.94). In Code Generation, the framework reached a Pass@1 score of 32.41, compared to 25.51 for the baseline.
Efficiency and Scalability
Despite the additional steps of reward scoring and iterative refinement, RewardDS remains computationally practical. The authors reported that the framework only added approximately 30% to the total server-side training time, making it a viable solution for real-world deployment where time and compute resources are constrained.
Privacy and Security Analysis
A key insight of the paper is that splitting the privacy budget between a generator and a reward model actually enhances security. By allocating a portion of the total privacy budget (epsilon) to each model, more noise is injected into the training process of each individual component. Empirical testing via Data Extraction and Membership Inference attacks showed that RewardDS is significantly less vulnerable to privacy breaches than single-model baselines.
Conclusion and Future Work
RewardDS represents a significant step forward in privacy-preserving LLM fine-tuning. By shifting the focus from text similarity to reward-driven quality, the framework allows organizations to leverage sensitive data without compromising user privacy. While current experiments utilized LoRA (Low-Rank Adaptation) due to hardware constraints, future work will explore full-parameter fine-tuning and the scalability of RewardDS on even larger LLM backbones. Additionally, the authors aim to further optimize client-side efficiency to reduce the computational burden on data owners.
Summary updated