Abstract
In task-oriented dialogue systems, generating natural language responses from structured meaning representations (MRs) is a critical task. However, the high cost of manual annotation often limits the availability of training data. While self-training—leveraging models to generate their own training data—is a common solution, it frequently introduces noise or uninformative examples. We propose SA$^2$, a two-phase self-augmentation framework. The first phase selects informative MRs based on predictive uncertainty, while the second phase refines response generation through latent representation aggregation. Our experiments on FewShotWOZ and FewShotSGD benchmarks demonstrate that SA$^2$ significantly outperforms existing self-training methods.
Introduction
Natural Language Generation (NLG) in task-oriented systems involves mapping Meaning Representations (MRs), such as inform(food=seafood, area=centre), into natural sentences. In "few-shot" scenarios where only a handful of examples are available, models struggle to capture domain-specific nuances.
Traditional self-training attempts to bridge this gap by using a fine-tuned model to label unlabeled data. However, this approach faces two primary hurdles:
- Noisy Data: Models may generate hallucinated slots or grammatically incorrect sentences.
- Uninformative Data: Models often select examples they are already confident in, which provides little new signal for learning.
The SA$^2$ Framework
To address these challenges, we introduce the Two-phase Self-augmentation (SA$^2$) framework. This method ensures the model learns from data that is both relevant to the domain and challenging enough to provide new information.
Phase I: Uncertainty-based Data Selection
The first phase focuses on identifying the most "informative" MRs from the unlabeled pool. We utilize Monte Carlo (MC) Dropout to estimate the model's uncertainty. By performing multiple forward passes with different dropout masks, we calculate two key metrics:
- Predictive Mean ($\mathbb{E}[p_\theta]$): This measures how "familiar" the model is with the input. A high mean suggests the data is in-domain and relevant.
- Predictive Variance ($Var[p_\theta]$): This measures the model's sensitivity or "uncertainty." High variance indicates the model has not yet mastered the specific patterns in that input.
The core innovation lies in selecting data from the "Goldilocks" zone: high mean and high variance. This targets examples that are relevant but difficult for the current model.
Phase II: Response Refinement
After selecting the optimal MRs, the system must generate high-quality target text. To mitigate the risk of the model overfitting to its own errors or hallucinations, SA$^2$ employs a refinement strategy. Instead of a single decoding pass, the model aggregates multiple perturbed latent representations. By averaging the hidden states from several passes (using dropout), the model produces a "smoothed" and more accurate response, which serves as the pseudo-label for the final fine-tuning stage.
Experimental Results
We evaluated SA$^2$ on two major benchmarks: FewShotWOZ (7 domains, ~50 examples/domain) and FewShotSGD (16 domains, ~35 examples/domain).
Performance Metrics
The proposed method consistently outperformed state-of-the-art baselines like SC-GPT and AUG-NLG:
- BLEU Scores: On the FewShotWOZ Restaurant domain, SA$^2$ achieved a BLEU score of 36.48, a significant improvement over the 34.62 achieved by SC-GPT.
- Slot Error Rate (ERR): SA$^2$ proved highly effective at reducing hallucinations. In the "Train" domain of FewShotWOZ, the error rate dropped from 1.90 to 1.62.
- Human Evaluation: In blind tests, human judges consistently rated SA$^2$ higher in both informativeness and naturalness compared to standard self-training variants.
Conclusion
The SA$^2$ framework demonstrates that the quality of synthetic data is more important than the quantity. By combining uncertainty-based selection with latent representation refinement, we can effectively train high-performing dialogue models even with extremely limited human-annotated data. While the computational cost of MC Dropout is a consideration, the gains in accuracy and reliability make it a compelling approach for few-shot NLG tasks.
Summary updated