Evaluating a dialog language generation system: comparing the mountain system to other NLG approaches

Brian Langner, Stephan Vogel, Alan W. BlackInterspeech 2010 · DOI

Abstract

This paper evaluates MOUNTAIN, a translation-based approach to Natural Language Generation (NLG) that treats structured data as a source language and natural language as the target. Using the Moses SMT engine on weather forecasting data, MOUNTAIN demonstrates competitive performance against handcrafted and grammar-based systems. Human evaluations reveal that MOUNTAIN's output is often more readable than the original human-written corpus due to a "regularization" effect. Furthermore, strong correlations between automatic metrics and human judgments validate the use of BLEU and METEOR in this domain.

Introduction

In spoken dialog systems, "tactical generation" is the process of converting a machine’s internal state—such as a database query result or a planned response—into a natural-sounding sentence. Traditionally, this process relied on hand-coded templates or complex grammar-based systems, both of which require significant linguistic expertise and development time.

This research addresses the need for a fully automatic, data-driven approach to NLG. The goal is to develop a system capable of learning to generate language simply by observing examples of data paired with human-written sentences, eliminating the need for manual rule-writing or extensive semantic tagging.

Methodology: The MOUNTAIN System

The researchers developed MOUNTAIN (a translation-based approach for natural language generation), which treats NLG as a machine translation task. In this framework, the "source language" consists of structured internal data states, and the "target language" is English.

System Architecture

The core of MOUNTAIN utilizes the Moses machine translation system, a standard tool for statistical machine translation (SMT). To bridge the gap between structured data and the translation engine, the authors developed a custom tokenization process. For instance, a raw data vector representing weather conditions, such as [1, W-SW, 10, 15], is transformed into a tokenized string like t1 W-SW 10 15.

Training and Generation

The system is trained on a parallel corpus to learn the statistical mapping between data tokens and human-written phrases. Once trained, MOUNTAIN can take novel, unseen data states and "translate" them into fluent sentences.

Experimental Setup

The system was evaluated using the SUMTIME-METEO corpus, which contains 465 entries of wind and precipitation data represented as 7-tuple numeric vectors. Each entry is paired with forecasts written by three different human weather forecasters. This dataset serves as a benchmark for comparing various NLG strategies, ranging from rule-based to statistical models.

Results and Discussion

MOUNTAIN was compared against 10 other NLG systems using both automatic metrics and human evaluation.

Human Evaluation and the "Regularization" Effect

In a 7-point Likert scale test, MOUNTAIN achieved a mean readability score of 4.63, outperforming the original human-written corpus (4.51) and all other SMT-based approaches.

Mean human ratings for clarity and readability across different NLG systems.

The authors hypothesize that this "regularization" effect occurs because the human corpus, written by multiple individuals, is inherently inconsistent. MOUNTAIN learns a consistent "average" style, resulting in output that feels more predictable and easier for readers to process.

Automatic Metrics and Validation

In terms of automatic performance, MOUNTAIN achieved a BLEU score of 0.5773, significantly outperforming other phrase-based statistical machine translation (PBSMT) baselines, which typically scored between 0.31 and 0.48. While it trailed behind complex, grammar-based systems (PSCFG), MOUNTAIN offers a significant advantage in development efficiency: it was trained in approximately one week, whereas the top-performing handcrafted system (SUMTIME-hybrid) required nearly a year to build.

Crucially, the study found a very high correlation (Pearson’s $r > 0.9$) between automatic metrics (BLEU, METEOR, NIST) and human judgments of clarity and readability.

Correlation between human judgments and automatic evaluation metrics.

This strong correlation suggests that for highly structured domains like weather forecasting, automatic metrics serve as a reliable proxy for human preference.

Conclusion

The MOUNTAIN system demonstrates that treating NLG as a translation problem is a viable and efficient alternative to handcrafted systems. However, limitations remain. The performance is highly sensitive to the design of the "source language" tokens, and the approach's scalability to more complex, open-ended dialog domains is yet to be tested. Future work will involve integrating MOUNTAIN into live, interactive spoken dialog systems to evaluate its performance in real-time user interactions.

Summary updated

BibTeX
@inproceedings{langner2010evaluating,
  author = {Brian Langner and Stephan Vogel and Alan W. Black},
  title = {{Evaluating a dialog language generation system: comparing the mountain system to other NLG approaches}},
  booktitle = {Interspeech},
  year = {2010},
  doi = {10.21437/interspeech.2010-353},
}
Figure