Abstract
Task-oriented dialogue systems often struggle with domain shifts between synthetic training data and real-world user interactions. While private user logs contain valuable signals for improvement, privacy concerns prevent direct access. This paper proposes a two-stage differentially private (DP) generation method that synthesizes training data by first modeling semantic parse distributions and then generating corresponding natural language utterances. Our approach significantly improves the linguistic and functional coverage of semantic parsers while maintaining formal privacy guarantees.
Introduction
Modern virtual assistants like Siri or Alexa process sensitive user information, necessitating strict privacy protocols. Developers are typically barred from inspecting real-world logs, leading to a "blind spot" regarding system failures or emerging user needs. While crowd-sourced data is often used as a proxy, it frequently fails to capture the nuances of authentic human-device interaction. Training directly on private data risks memorization and potential leakage of sensitive sequences. The core challenge lies in leveraging private data signals to improve system performance without compromising user privacy.
Methodology: Two-Stage DP Generation
To address this, we introduce a two-stage Differentially Private (DP) generation framework. Unlike standard 1-stage baselines that attempt to model user utterances directly, our method decomposes the task into functional intent and linguistic expression.
- DP Parse Generation: A model is trained using DP-SGD (Differentially Private Stochastic Gradient Descent) to learn the distribution of semantic parses (code-like intent representations) from private data.
- DP Parse-to-Utterance: A second DP model is trained to generate natural language utterances conditioned on these parse trees.
This separation allows the system to explicitly capture the "what" (intent) before the "how" (expression). Privacy is ensured via DP-SGD, which utilizes gradient clipping and Gaussian noise to provide a mathematical privacy budget ($\epsilon$). Due to the post-processing property of Differential Privacy, synthetic data sampled from these models can be used freely for downstream training without further privacy risk.
Experimental Results
We evaluated our approach on the SMCalFlow v2.0 and TreeDST benchmarks. Our 2-stage method consistently outperformed the 1-stage baseline across various privacy budgets.
Key Findings
- Enhanced Semantic Coverage: The 2-stage approach improved the capture of rare API calls by 1.3x. It achieved 80% coverage of the top-25 most common function types in private data, compared to 64% for the baseline.
- Improved Synthesis Quality: MAUVE scores, which measure the similarity between synthetic and real text distributions, improved by 2.5x over the 1-stage baseline.
- Effective Domain Adaptation: In scenarios where new features (e.g., "Weather") appeared only in private data, our method improved parser accuracy on those features by 13.4% over the baseline.
- Budget Allocation: Optimal performance was achieved by allocating more training epochs to the second stage (text generation) rather than the first (parse generation). Specifically, an 8-epoch split for the second stage was more effective than a balanced split.
Conclusion
Decomposing the generation of synthetic training data into semantic and linguistic stages provides a robust path for privacy-preserving domain adaptation. By "forcing" the model to learn functional distributions first, we achieve significantly better coverage of user intents. Future work will explore active learning strategies to synthesize data where the parser is most uncertain, further optimizing the utility of the privacy budget.
Summary updated