TL;DR
This paper explores how to improve structure-to-text generation, particularly in low-data scenarios, by combining self-training with pre-trained language models and acceptability classifiers. They found that incorporating dialog context significantly improves model alignment with user language and self-consistency, while self-training shows promise for semantic correctness in low-data settings.
What Problem Does This Paper Address?
Natural Language Generation (NLG) systems are becoming increasingly prevalent, but training reliable and high-quality models typically requires vast amounts of data. This is a significant bottleneck, especially when adapting models to new domains or languages. The paper addresses this challenge by investigating methods to improve NLG performance with limited training data, focusing on techniques that enhance semantic correctness and fluency. They also explore how to make NLG systems better at adapting to user language and maintaining conversational coherence.
Methods
The authors employ a combination of three key techniques:
- Context-Conditioning: For task-oriented dialog systems, they investigate the impact of including previous turns of conversation as input to the NLG model. This allows the model to better understand the ongoing dialog and generate more contextually relevant responses. They also experimented with "templatized inputs," which convert structured dialog acts into semi-natural text to guide the generation process.
- Self-Training: This semi-supervised learning technique leverages unlabeled data to improve model performance. The authors generate synthetic unlabeled data by deleting parts of the structured input query. They then use a generation model to produce pseudo-labels for this unlabeled data and a reconstruction model to verify the generated inputs, iteratively refining the model.
- Acceptability Classifiers: To filter out potentially incorrect or ungrammatical generated responses, they train classifiers that predict whether a response is semantically accurate and grammatically sound. These classifiers are trained on synthetic acceptable/unacceptable samples generated using a MaskFilling strategy and validated using entailment and BLEU scores.
Datasets
The paper uses three datasets from the GEM Shared Task:
- Schema-Guided Dialog (SGD): A dataset for task-oriented dialog systems.
- E2E: A dataset for end-to-end NLG.
- WebNLG-en: A dataset for generating text from structured data.
The authors note that these datasets were chosen because they closely resemble natural language generation in a task-oriented dialog setting.
Key Insights and Ideas
- Context is King (for Dialog): Including multiple turns of dialog context significantly boosts NLG performance, leading to better alignment with user language and more self-consistent responses. This suggests that understanding the conversational history is crucial for natural-sounding dialog generation.
- Self-Training for Semantic Fidelity: While self-training didn't dramatically improve BLEU scores in full-data settings, it showed a sharp increase in exact reconstruction match rates, particularly in lower-data regimes. This indicates its potential for improving the semantic accuracy of generated text when labeled data is scarce.
- Acceptability Classifiers as a Quality Filter: Acceptability classifiers can help select better responses from a set of generated candidates, especially in scenarios where the generation model might produce fluent but semantically incorrect outputs.
- The Value of Pre-trained Models: The paper builds upon the success of pre-trained language models like BART and GPT-2, demonstrating how these can be effectively adapted for structure-to-text generation with the proposed techniques.
Key Results
- Context-Conditioning: Including dialog context led to substantial gains in BLEU scores on the Schema-Guided Dialog dataset. Specifically, adding the prompt improved BLEU by over 3 points, and adding four more turns of context improved it by an additional 5 points.
- Self-Training: While BLEU scores showed only marginal improvement with self-training in full-data settings, the exact reconstruction match rate on validation sets saw a sharp increase, especially in lower data regimes (Table 5).
- Acceptability Classifiers: Filtering with acceptability classifiers resulted in only slight changes to BLEU scores and exact reconstruction match rates, but the authors observed a slight increase in the number of paraphrases identified as better by the classifier.
Limitations and Future Work
- Low-Data Gains: While self-training shows promise for semantic correctness in low-data settings, it hasn't yet brought systems fully in line with high-data trained systems. The authors suggest future work could focus on improving these gains.
- Few-Shot Tracks: The authors encourage future GEM challenges to include few-shot tracks to promote research on data efficiency. They also suggest making unpaired inputs available for these tracks to facilitate research on generalizing to unseen or few-shot test items.
- Reconstruction Model Issues: The paper notes that the reconstruction model sometimes struggled with elliptical constructions and inadequate normalization (e.g., interchangeable values for slots). Future work could address this by comparing normalized slot values rather than just exact string matches.
- Human Evaluation: The authors suggest that verifying the enhanced naturalness and user perception of context-enhanced NLG would require human evaluation in actual dialogs.
Summary updated