Introduction
Task-oriented dialogue systems, such as virtual assistants, rely on semantic parsing to translate natural language user utterances into executable code or API calls. Improving these systems typically requires access to real-world user data to capture the nuances of how people actually communicate. However, this data is often sensitive, creating a "privacy barrier" for developers.
Currently, developers often resort to using synthetic datasets created by crowd-workers. This leads to a significant domain shift, as the language used by crowd-workers frequently differs from that of actual users. While Differential Privacy (DP) can be used to train models on private data, standard approaches often fail to provide developers with the inspectable data needed to identify missing features or functional gaps. This research introduces a method to generate high-quality, inspectable synthetic data that preserves the diversity of private usage while maintaining rigorous privacy guarantees.
Methodology: The 2-Stage DP Generation Method
The researchers propose a novel two-stage approach for generating synthetic training data, contrasting it with a standard one-stage baseline.
- The 1-Stage Baseline: This approach involves fine-tuning a pre-trained language model (like GPT-2) directly on private user utterances using DP-SGD (Differentially Private Stochastic Gradient Descent). While this protects individual privacy by adding noise during training, it often struggles to capture the full breadth of user intents.
- The 2-Stage Proposed Method: This method explicitly leverages the structured nature of semantic parses by splitting the generation task into two distinct components:
- Stage 1 (Parse Generation): A model is DP-trained to generate the underlying semantic parses (the "code") found in the private data.
- Stage 2 (Parse-to-Utterance): A separate model is DP-trained to translate these semantic parses back into natural language utterances.
By decoupling "what" users are doing from "how" they say it, the 2-stage method ensures that even rare or new functionalities are better represented in the synthetic output.
Experimental Setup
The method was evaluated on two major conversational semantic parsing datasets:
- SMCalFlow v2.0: Focuses on calendar, weather, and location-based queries.
- TreeDST: Covers 10 diverse domains (e.g., flights, hotels) using hierarchical semantic representations.
To test domain adaptation, the researchers simulated a "low-resource" scenario where specific features (like "Weather") were missing from the public training data but present in the private set. The goal was to see how effectively the model could "re-learn" these features from private data without compromising privacy.
Key Results and Insights
The 2-stage method demonstrated significant improvements over the 1-stage baseline across multiple dimensions:
- Enhanced Text Quality: The 2-stage approach achieved MAUVE scores (measuring similarity to human text) up to 2.5x higher than the baseline.
- Superior Semantic Coverage: It improved the overlap of parse tree function types by 1.3x, indicating that the synthetic data more accurately reflected the actual API calls users were making.
- Downstream Performance: In domain adaptation tasks, the 2-stage method led to an 8.5% point increase in accuracy for newly added features compared to a non-augmented parser.
- Privacy Efficiency: The 2-stage method remained robust even under strict privacy budgets (low $\epsilon$), proving to be a more efficient way to utilize the "privacy budget."
A key insight from the study is that 1-stage DP models tend to be overly "conservative," gravitating toward common patterns and losing the "long tail" of diverse requests due to the noise added by DP-SGD. By explicitly modeling the semantic structure, the 2-stage approach forces the preservation of functional diversity.
Limitations and Future Directions
While promising, the approach has several limitations. DP-SGD is computationally intensive, requiring significant memory and time. The method also relies on an initial "low-resource" parser to generate noisy labels for the private data; if this initial parser is too weak, the synthetic data quality may suffer. Furthermore, while DP protects individuals, it may still reflect common patterns shared by many users. Future work may explore Active Learning to strategically select the most useful private utterances for synthesis, further optimizing the privacy-utility tradeoff.
Summary updated