Char2char Generation with Reranking for the E2E NLG Challenge

Shubham Agarwal, Marc Dymetman, Éric Gaussier

Abstract: Traditional Natural Language Generation (NLG) systems often rely on complex word-level preprocessing and delexicalization to handle diverse vocabularies. This paper explores a character-level sequence-to-sequence approach for the E2E NLG Challenge, demonstrating that treating text as a raw character stream can simplify the pipeline while maintaining high performance. We further enhance the model's semantic adequacy through reranking strategies, including a reverse model and a classifier trained on synthetic error data. Our results indicate that character-level models are robust, sound natural to humans, and rank among the top-performing systems in the challenge.

Introduction

Natural Language Generation (NLG) typically involves transforming structured Meaning Representations (MRs) into human-readable text. Standard word-level models often struggle with "Out-of-Vocabulary" (OOV) words, necessitating delexicalization—a process of replacing specific entities with placeholders. However, delexicalization is brittle, often ignoring linguistic nuances like gender or number agreement. This work investigates whether a character-level model can bypass these issues by learning to generate text character-by-character, effectively "copying" names and values directly from the input.

Methodology

The proposed system follows a three-stage pipeline designed to maximize both fluency and semantic accuracy:

  1. Forward Model: A vanilla character-level sequence-to-sequence RNN with an attention mechanism. It processes raw MR strings without tokenization, lowercasing, or delexicalization.
  2. Synthetic Data Generation: To improve semantic accuracy, the authors created a synthetic dataset by perturbing existing MR-sentence pairs. By adding or removing slots in the MR, they generated examples of "omission" and "addition" errors to train diagnostic tools.
  3. Reranking: A "top-n" list of candidate sentences is generated via beam search. Two reranking strategies are employed:
  4. Reverse Model: A model trained to translate generated sentences back into MRs. High-quality candidates are those whose reconstructed MR matches the original input.
  5. Classifier Reranker: A logistic regression classifier trained on synthetic data to detect information gaps or hallucinations.
System architecture showing the forward model, reverse model, and reranking pipeline.

Experiments and Results

The models were evaluated on the E2E NLG Challenge dataset, which contains approximately 50,000 restaurant-related MR-reference pairs. The character-level approach proved remarkably robust. Because the vocabulary is limited to characters and punctuation, the model never encounters unknown words.

Interestingly, the authors observed an inverse relationship between BLEU scores and semantic adequacy. While the "Primary" submission (using the classifier reranker) was more factually accurate, the "Forward" (vanilla) model achieved a higher BLEU score, ranking 4th overall in the challenge. This suggests that while reranking improves accuracy, it may slightly reduce the stylistic overlap measured by BLEU.

Performance of different model variations on the E2E NLG Challenge leaderboard.

Human evaluations for "Quality" and "Naturalness" placed the system in the second-best cluster among all competitors, confirming that character-level generation produces fluent, human-like output that is indistinguishable from word-level systems to the end-user.

Discussion and Conclusion

The primary challenge for character-level models remains complex MRs with many slots (7 or 8), where the model may omit information or "hallucinate" common values. The authors suggest that the reranking scores developed in this work could be integrated into a Reinforcement Learning (RL) framework. By using these scores as reward signals during training, the model could be optimized directly for semantic accuracy alongside linguistic fluency.

In conclusion, this study demonstrates that character-level NLG is a viable and effective alternative to word-level systems. By reducing the need for brittle preprocessing and employing targeted reranking strategies, character-level models can achieve state-of-the-art results in complex generation tasks.

Summary updated

Abstract

This paper describes our submission to the E2E NLG Challenge. Recently, neural seq2seq approaches have become mainstream in NLG, often resorting to pre- (respectively post-) processing delexicalization (relexicalization) steps at the word-level to handle rare words. By contrast, we train a simple character level seq2seq model, which requires no pre/post-processing (delexicalization, tokenization or even lowercasing), with surprisingly good results. For further improvement, we explore two re-ranking approaches for scoring candidates. We also introduce a synthetic dataset creation procedure, which opens up a new way of creating artificial datasets for Natural Language Generation.
BibTeX
@inproceedings{agarwal2018char,
  author = {Shubham Agarwal and Marc Dymetman and Éric Gaussier},
  title = {{Char2char Generation with Reranking for the E2E NLG Challenge}},
  booktitle = {INLG},
  year = {2018},
  doi = {10.18653/v1/w18-6555},
}
Figure