Privacy-Preserving Domain Adaptation of Semantic Parsers

Fatemehsadat Mireshghallah, Richard Shin, Yu Su, Tatsunori Hashimoto, Jason Eisner

Abstract

Task-oriented dialogue systems often struggle with domain shifts between synthetic training data and real-world user interactions. While private user logs contain valuable signals for improvement, privacy concerns prevent direct access. This paper proposes a two-stage differentially private (DP) generation method that synthesizes training data by first modeling semantic parse distributions and then generating corresponding natural language utterances. Our approach significantly improves the linguistic and functional coverage of semantic parsers while maintaining formal privacy guarantees.

Introduction

Modern virtual assistants like Siri or Alexa process sensitive user information, necessitating strict privacy protocols. Developers are typically barred from inspecting real-world logs, leading to a "blind spot" regarding system failures or emerging user needs. While crowd-sourced data is often used as a proxy, it frequently fails to capture the nuances of authentic human-device interaction. Training directly on private data risks memorization and potential leakage of sensitive sequences. The core challenge lies in leveraging private data signals to improve system performance without compromising user privacy.

Methodology: Two-Stage DP Generation

To address this, we introduce a two-stage Differentially Private (DP) generation framework. Unlike standard 1-stage baselines that attempt to model user utterances directly, our method decomposes the task into functional intent and linguistic expression.

The proposed 2-stage DP generation method decomposes the task into parse generation and conditional utterance synthesis.
  1. DP Parse Generation: A model is trained using DP-SGD (Differentially Private Stochastic Gradient Descent) to learn the distribution of semantic parses (code-like intent representations) from private data.
  2. DP Parse-to-Utterance: A second DP model is trained to generate natural language utterances conditioned on these parse trees.

This separation allows the system to explicitly capture the "what" (intent) before the "how" (expression). Privacy is ensured via DP-SGD, which utilizes gradient clipping and Gaussian noise to provide a mathematical privacy budget ($\epsilon$). Due to the post-processing property of Differential Privacy, synthetic data sampled from these models can be used freely for downstream training without further privacy risk.

Experimental Results

We evaluated our approach on the SMCalFlow v2.0 and TreeDST benchmarks. Our 2-stage method consistently outperformed the 1-stage baseline across various privacy budgets.

Quantitative performance comparison across benchmarks showing the superiority of the 2-stage method.

Key Findings

  • Enhanced Semantic Coverage: The 2-stage approach improved the capture of rare API calls by 1.3x. It achieved 80% coverage of the top-25 most common function types in private data, compared to 64% for the baseline.
  • Improved Synthesis Quality: MAUVE scores, which measure the similarity between synthetic and real text distributions, improved by 2.5x over the 1-stage baseline.
  • Effective Domain Adaptation: In scenarios where new features (e.g., "Weather") appeared only in private data, our method improved parser accuracy on those features by 13.4% over the baseline.
  • Budget Allocation: Optimal performance was achieved by allocating more training epochs to the second stage (text generation) rather than the first (parse generation). Specifically, an 8-epoch split for the second stage was more effective than a balanced split.

Conclusion

Decomposing the generation of synthetic training data into semantic and linguistic stages provides a robust path for privacy-preserving domain adaptation. By "forcing" the model to learn functional distributions first, we achieve significantly better coverage of user intents. Future work will explore active learning strategies to synthesize data where the parser is most uncertain, further optimizing the utility of the privacy budget.

Summary updated

Abstract

Task-oriented dialogue systems often assist users with personal or confidential matters. For this reason, the developers of such a system are generally prohibited from observing actual usage. So how can they know where the system is failing and needs more training data or new functionality? In this work, we study ways in which realistic user utterances can be generated synthetically, to help increase the linguistic and functional coverage of the system, without compromising the privacy of actual users. To this end, we propose a two-stage Differentially Private (DP) generation method which first generates latent semantic parses, and then generates utterances based on the parses. Our proposed approach improves MAUVE by 2.5$\times$ and parse tree function type overlap by 1.3$\times$ relative to current approaches for private synthetic data generation, improving both on fluency and semantic coverage. We further validate our approach on a realistic domain adaptation task of adding new functionality from private user data to a semantic parser, and show overall gains of 8.5% points in accuracy with the new feature.
BibTeX
@inproceedings{mireshghallah2023privacy,
  author = {Fatemehsadat Mireshghallah and Richard Shin and Yu Su and Tatsunori Hashimoto and Jason Eisner},
  title = {{Privacy-Preserving Domain Adaptation of Semantic Parsers}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.271},
}
Figure