Abstract: Traditional Natural Language Generation (NLG) systems often rely on complex word-level preprocessing and delexicalization to handle diverse vocabularies. This paper explores a character-level sequence-to-sequence approach for the E2E NLG Challenge, demonstrating that treating text as a raw character stream can simplify the pipeline while maintaining high performance. We further enhance the model's semantic adequacy through reranking strategies, including a reverse model and a classifier trained on synthetic error data. Our results indicate that character-level models are robust, sound natural to humans, and rank among the top-performing systems in the challenge.
Introduction
Natural Language Generation (NLG) typically involves transforming structured Meaning Representations (MRs) into human-readable text. Standard word-level models often struggle with "Out-of-Vocabulary" (OOV) words, necessitating delexicalization—a process of replacing specific entities with placeholders. However, delexicalization is brittle, often ignoring linguistic nuances like gender or number agreement. This work investigates whether a character-level model can bypass these issues by learning to generate text character-by-character, effectively "copying" names and values directly from the input.
Methodology
The proposed system follows a three-stage pipeline designed to maximize both fluency and semantic accuracy:
- Forward Model: A vanilla character-level sequence-to-sequence RNN with an attention mechanism. It processes raw MR strings without tokenization, lowercasing, or delexicalization.
- Synthetic Data Generation: To improve semantic accuracy, the authors created a synthetic dataset by perturbing existing MR-sentence pairs. By adding or removing slots in the MR, they generated examples of "omission" and "addition" errors to train diagnostic tools.
- Reranking: A "top-n" list of candidate sentences is generated via beam search. Two reranking strategies are employed:
- Reverse Model: A model trained to translate generated sentences back into MRs. High-quality candidates are those whose reconstructed MR matches the original input.
- Classifier Reranker: A logistic regression classifier trained on synthetic data to detect information gaps or hallucinations.
Experiments and Results
The models were evaluated on the E2E NLG Challenge dataset, which contains approximately 50,000 restaurant-related MR-reference pairs. The character-level approach proved remarkably robust. Because the vocabulary is limited to characters and punctuation, the model never encounters unknown words.
Interestingly, the authors observed an inverse relationship between BLEU scores and semantic adequacy. While the "Primary" submission (using the classifier reranker) was more factually accurate, the "Forward" (vanilla) model achieved a higher BLEU score, ranking 4th overall in the challenge. This suggests that while reranking improves accuracy, it may slightly reduce the stylistic overlap measured by BLEU.
Human evaluations for "Quality" and "Naturalness" placed the system in the second-best cluster among all competitors, confirming that character-level generation produces fluent, human-like output that is indistinguishable from word-level systems to the end-user.
Discussion and Conclusion
The primary challenge for character-level models remains complex MRs with many slots (7 or 8), where the model may omit information or "hallucinate" common values. The authors suggest that the reranking scores developed in this work could be integrated into a Reinforcement Learning (RL) framework. By using these scores as reward signals during training, the model could be optimized directly for semantic accuracy alongside linguistic fluency.
In conclusion, this study demonstrates that character-level NLG is a viable and effective alternative to word-level systems. By reducing the need for brittle preprocessing and employing targeted reranking strategies, character-level models can achieve state-of-the-art results in complex generation tasks.
Summary updated