Statistical Acquisition of Content Selection Rules for Natural Language Generation

Pablo Ariel Duboué, Kathleen R. McKeownEMNLP 2003 · OpenReview

Introduction

In Natural Language Generation (NLG), the "content selection" phase—deciding what information to include from a large knowledge base—is a critical first step. Traditionally, this process relied on manually crafted rules, which are difficult to scale and maintain across different domains. This paper presents an automated statistical pipeline that learns these selection rules by aligning structured semantic data with existing human-written texts.

The Automated Pipeline

The authors propose a system that bridges the gap between structured frames/triples and unstructured natural language. The architecture consists of several stages, from initial data matching to the induction of complex logic.

Figure 1: The system architecture for learning content selection rules, showing the flow from semantic inputs and target texts to the generation of baseline, class-based, and content selection rules.

The process begins with Exact Matching, identifying data points like names or dates that appear verbatim in the target text. For more abstract information, the system employs Statistical Selection. By clustering semantic data (e.g., grouping "actors" vs. "directors") and analyzing the bi-gram language models of their associated biographies, the system uses cross-entropy and Mann-Whitney U tests to determine which data fields significantly influence the resulting text. Finally, the RIPPER algorithm is used for Rule Induction to capture fine-grained conditional logic.

Data and Evaluation

The study utilized a custom corpus of 1,100 celebrity fact-sheets from E! Online (containing over 100k triples) paired with biographies from biography.com and imdb.com. A "gold standard" subset was hand-tagged to evaluate the system's performance.

Experimental Results

The evaluation focused on the system's ability to act as a high-recall filter, identifying relevant information while discarding noise.

Figure 4: Performance metrics for different rule types across development and imdb.com datasets. The "class-based" rules achieve the highest recall (0.94) while significantly reducing the amount of selected data compared to the "select-all" baseline.

As shown in the results table, the "class-based" statistical rules achieved a remarkable 94% recall on the development set. Crucially, the system filtered out approximately 50% of the input data (selecting 550 items vs. 1129 in the "select-all" baseline), effectively reducing the search space for subsequent generation stages. While the RIPPER-induced "content-selection" rules provided higher precision and human-readable logic, their lower recall makes them less suitable as a primary filter compared to the broader statistical approach.

Conclusion and Future Directions

The research demonstrates that language can serve as a reliable proxy for information importance. By identifying the "fingerprints" that specific data fields leave on vocabulary, NLG systems can automatically adapt to new genres. Future work aims to address limitations regarding aggregated information (e.g., summarizing multiple award wins) and incorporate more complex rhetorical goals through techniques like spreading activation in knowledge graphs.

Summary updated

Abstract

A Natural Language Generation system produces text using as input semantic data. One of its very first tasks is to decide which pieces of information to convey in the output. This task, called Content Selection, is quite domain dependent, requiring considerable re-engineering to transport the system from one scenario to another. In this paper, we present a method to acquire content selection rules automatically from a corpus of text and associated semantics. Our proposed technique was evaluated by comparing its output with information selected by human authors in unseen texts, where we were able to filter half the input data set without loss of recall.
BibTeX
@inproceedings{duboue2003statistical,
  author = {Pablo Ariel Duboué and Kathleen R. McKeown},
  title = {{Statistical Acquisition of Content Selection Rules for Natural Language Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2003},
  url = {https://openreview.net/forum?id=S14NPGfO-H},
}
Figure