Privacy-Preserving Domain Adaptation of Semantic Parsers

Fatemehsadat Mireshghallah, Yu Su, Tatsunori Hashimoto, Jason Eisner, Richard Shin

Introduction

Task-oriented dialogue systems, such as virtual assistants, rely on semantic parsing to translate natural language user utterances into executable code or API calls. Improving these systems typically requires access to real-world user data to capture the nuances of how people actually communicate. However, this data is often sensitive, creating a "privacy barrier" for developers.

Currently, developers often resort to using synthetic datasets created by crowd-workers. This leads to a significant domain shift, as the language used by crowd-workers frequently differs from that of actual users. While Differential Privacy (DP) can be used to train models on private data, standard approaches often fail to provide developers with the inspectable data needed to identify missing features or functional gaps. This research introduces a method to generate high-quality, inspectable synthetic data that preserves the diversity of private usage while maintaining rigorous privacy guarantees.

Methodology: The 2-Stage DP Generation Method

The researchers propose a novel two-stage approach for generating synthetic training data, contrasting it with a standard one-stage baseline.

  1. The 1-Stage Baseline: This approach involves fine-tuning a pre-trained language model (like GPT-2) directly on private user utterances using DP-SGD (Differentially Private Stochastic Gradient Descent). While this protects individual privacy by adding noise during training, it often struggles to capture the full breadth of user intents.
  2. The 2-Stage Proposed Method: This method explicitly leverages the structured nature of semantic parses by splitting the generation task into two distinct components:
  3. Stage 1 (Parse Generation): A model is DP-trained to generate the underlying semantic parses (the "code") found in the private data.
  4. Stage 2 (Parse-to-Utterance): A separate model is DP-trained to translate these semantic parses back into natural language utterances.
The 2-stage DP generation method separately models parse trees and utterances to preserve functional diversity.

By decoupling "what" users are doing from "how" they say it, the 2-stage method ensures that even rare or new functionalities are better represented in the synthetic output.

Experimental Setup

The method was evaluated on two major conversational semantic parsing datasets:

  • SMCalFlow v2.0: Focuses on calendar, weather, and location-based queries.
  • TreeDST: Covers 10 diverse domains (e.g., flights, hotels) using hierarchical semantic representations.

To test domain adaptation, the researchers simulated a "low-resource" scenario where specific features (like "Weather") were missing from the public training data but present in the private set. The goal was to see how effectively the model could "re-learn" these features from private data without compromising privacy.

Key Results and Insights

The 2-stage method demonstrated significant improvements over the 1-stage baseline across multiple dimensions:

  • Enhanced Text Quality: The 2-stage approach achieved MAUVE scores (measuring similarity to human text) up to 2.5x higher than the baseline.
  • Superior Semantic Coverage: It improved the overlap of parse tree function types by 1.3x, indicating that the synthetic data more accurately reflected the actual API calls users were making.
  • Downstream Performance: In domain adaptation tasks, the 2-stage method led to an 8.5% point increase in accuracy for newly added features compared to a non-augmented parser.
  • Privacy Efficiency: The 2-stage method remained robust even under strict privacy budgets (low $\epsilon$), proving to be a more efficient way to utilize the "privacy budget."
Performance comparison of 1-stage and 2-stage methods across SMCalFlow and TreeDST datasets at various privacy budgets (ε).

A key insight from the study is that 1-stage DP models tend to be overly "conservative," gravitating toward common patterns and losing the "long tail" of diverse requests due to the noise added by DP-SGD. By explicitly modeling the semantic structure, the 2-stage approach forces the preservation of functional diversity.

Limitations and Future Directions

While promising, the approach has several limitations. DP-SGD is computationally intensive, requiring significant memory and time. The method also relies on an initial "low-resource" parser to generate noisy labels for the private data; if this initial parser is too weak, the synthetic data quality may suffer. Furthermore, while DP protects individuals, it may still reflect common patterns shared by many users. Future work may explore Active Learning to strategically select the most useful private utterances for synthesis, further optimizing the privacy-utility tradeoff.

Summary updated

Abstract

Task-oriented dialogue systems often assist users with personal or confidential matters. For this reason, the developers of such a system are generally prohibited from observing actual usage. So how can they know where the system is failing and needs more training data or new functionality? In this work, we study ways in which realistic user utterances can be generated synthetically, to help increase the linguistic and functional coverage of the system, without compromising the privacy of actual users. To this end, we propose a two-stage Differentially Private (DP) generation method which first generates latent semantic parses, and then generates utterances based on the parses. Our proposed approach improves MAUVE by 2.5$\times$ and parse tree function type overlap by 1.3$\times$ relative to current approaches for private synthetic data generation, improving both on fluency and semantic coverage. We further validate our approach on a realistic domain adaptation task of adding new functionality from private user data to a semantic parser, and show overall gains of 8.5% points in accuracy with the new feature.
BibTeX
@inproceedings{mireshghallah2023privacy,
  author = {Fatemehsadat Mireshghallah and Yu Su and Tatsunori Hashimoto and Jason Eisner and Richard Shin},
  title = {{Privacy-Preserving Domain Adaptation of Semantic Parsers}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.271},
}
Figure