E2E NLG Challenge: Neural Models vs. Templates

Yevgeniy Puzikov, Iryna GurevychINLG 2018 · DOI

Abstract

This paper evaluates the performance of neural encoder-decoder models against rule-based template systems in the context of the E2E Natural Language Generation (NLG) Challenge. While neural models achieve higher scores on automatic metrics like BLEU, they often inherit noise from crowd-sourced training data, leading to factual omissions and grammatical errors. In contrast, a simple template-based system provides 100% reliability and competitive human evaluation scores, highlighting the limitations of current end-to-end data-driven approaches.

Introduction

Natural Language Generation (NLG) involves converting structured data, such as restaurant attributes, into fluent human-readable text. The E2E NLG Challenge was established to advance end-to-end data-driven methods that learn this mapping directly from data. However, these models are highly sensitive to the quality of training data. The authors observed that crowd-sourced datasets often contain "noise"—discrepancies where references omit key information or include extraneous details. This study investigates whether the complexity of neural models is justified or if a simple template-based system can offer superior performance in real-world scenarios.

Methods

The authors developed two distinct systems to compare these paradigms:

Model-D (Data-driven)

Model-D is a neural encoder-decoder architecture. Unlike standard models that treat input as a sequence, Model-D exploits the fixed structure of the input attributes. Each attribute is assigned a "positional ID" (e.g., name is always position 6), enabling a more efficient, non-sequential encoding process. The architecture incorporates an embedding matrix, a dense hidden layer for encoding, and a GRU-based decoder with an attention mechanism.

Architecture of Model-D, featuring a non-sequential encoder using positional IDs.

Model-T (Template-based)

Model-T is a deterministic system built in just a few hours. It utilizes a "master template" (e.g., [name] is a [eatType]...) and a set of hand-written rules to manage variations, such as phrasing numerical ratings or omitting missing attributes.

Dataset Analysis

The study utilizes the E2E NLG Dataset, containing over 50,000 instances in the restaurant domain. Each instance pairs a Meaning Representation (MR) with multiple human-written references. A manual analysis by the authors revealed that approximately 40% of the dataset contained discrepancies, such as omitted or modified content. This noise significantly impacts the learning process for neural models.

Key Results and Discussion

The evaluation revealed a stark contrast between automatic metrics and linguistic reliability.

The Metric Trap

Model-D outperformed the TGen baseline across all automatic metrics, achieving a BLEU score of 0.7128. However, these metrics reward models that closely mimic the training data. Because the training data was noisy, Model-D "learned" to drop content, failing to provide factual consistency.

Reliability vs. Diversity

In a manual audit of 100 samples, Model-T exhibited zero errors in grammar or content. Conversely, Model-D dropped information in 49 instances and produced 12 punctuation errors. While neural models offer more varied phrasing, they lack the predictability required for many real-world applications.

Comparison of error rates across models, demonstrating the superior reliability of the template-based Model-T.

Human Evaluation

Despite lower automatic scores, Model-T was ranked in the second-best cluster for both quality and naturalness in the final challenge standings. This indicates that human evaluators prioritize grammatical correctness and factual accuracy over lexical variety.

Conclusion

The paper serves as a cautionary tale for the NLG community. While end-to-end neural architectures are powerful, they are currently limited by the quality of available training data. Future research should focus on reformulating loss functions to better handle noisy references. Ultimately, the transparency and reliability of symbolic methods like Model-T remain highly competitive, especially when data quality cannot be guaranteed.

Summary updated

BibTeX
@inproceedings{puzikov2018e,
  author = {Yevgeniy Puzikov and Iryna Gurevych},
  title = {{E2E NLG Challenge: Neural Models vs. Templates}},
  booktitle = {INLG},
  year = {2018},
  doi = {10.18653/v1/w18-6557},
}

Loading related content...

Figure