Experimenting with the Interaction between Aggregation and Text Structuring

Hua ChengNAACL 2000

TL;DR

This paper tackles the challenge of integrating text aggregation (combining information) with text planning in natural language generation (NLG) systems. By framing these interactions as preferences within a Genetic Algorithm, the authors demonstrate that capturing these preferences is crucial for generating coherent and high-quality text, moving beyond traditional pipeline architectures.

What Problem Does This Paper Address?

Traditional Natural Language Generation (NLG) systems often employ a pipeline architecture, where different stages like content selection, text planning, and surface realization operate sequentially. While this modularity simplifies design, it struggles to capture the complex interdependencies between these stages. One such critical interaction is between text aggregation, which condenses information, and text planning, which structures the discourse and maintains coherence. Aggregation can disrupt the flow of information by removing facts, potentially leading to less coherent text. The paper questions where and how aggregation should occur and how it relates to other planning processes, particularly maintaining local coherence (e.g., smooth transitions between sentences). The authors aim to explore the preferences that govern these interactions and show how incorporating them can lead to better generated text.

Methods

The authors propose using a Genetic Algorithm (GA) as the core of their text planner. This approach is well-suited because the search space for text planning is vast and not necessarily smooth, and a GA can explore many combinations to find good solutions without requiring a global optimum.

Their novel contribution lies in how they integrate preferences for aggregation and local coherence into the GA's evaluation function. They define specific heuristics that capture these preferences:

  • Global Coherence: Preferences for different types of semantic relations (e.g., preferring a direct semantic relation over a Conjunct or Joint relation).
  • Local Coherence: Preferences for specific center transitions (e.g., Continuation > Associate shifting > Retaining > Shifting > Resuming), based on Centering Theory and observations of museum descriptions.
  • Embedding Preferences: Defining criteria for "good," "normal," and "bad" embeddings, considering factors like the type of referring expression, the syntactic realization of the embedded part, and its position within the text structure.

These heuristics are then used to score potential text structures generated by the GA. The GA iteratively refines sequences of facts and their relations, guided by this evaluation function, to produce coherent text.

Datasets

The paper does not explicitly mention training datasets. For evaluation, they used four human-written museum descriptions. These descriptions were manually broken down into individual facts and relations to serve as input for their system. They also generated texts using their system and compared the scores assigned by their evaluation function to those of human-written texts.

Key Insights and Ideas

  • Preferences, Not Absolute Rules: The core idea is that the relative preferences among different generation factors (like aggregation and coherence) are more critical for text quality than absolute rules for each factor in isolation.
  • GA as an Integrator: A Genetic Algorithm provides a flexible framework to simultaneously consider and balance these competing preferences, which is difficult in a strict pipeline.
  • Embedding is Tricky: Aggregation, specifically embedding one clause into an NP of another, can easily disrupt local coherence. The paper highlights the need for careful consideration of embedding's impact on discourse flow.
  • Museum Descriptions as a Testbed: The choice of museum descriptions is insightful, as they often involve describing objects, their features, and historical context, naturally presenting challenges for aggregation and coherence.

Key Results

The authors implemented their GA-based planner with the defined heuristics and evaluated its output.

  • Qualitative Success: The system was able to generate texts of "similar qualities to human written texts."
  • Evaluation Function Validation: They developed an evaluation function based on their heuristics and showed that it correlated highly (.9567 Pearson correlation) with a second, independently derived rater. This suggests their heuristics effectively capture what makes a text coherent.
  • "Nearly Best Texts": Their GA system, when run on the input derived from human texts, achieved scores comparable to or even higher than the original human texts, suggesting the evaluation function can identify good text structures.

Limitations and Future Work

  • Limited Scope of Embedding: The paper focused on a specific type of embedding (shifting a clause into an NP). The authors acknowledge that other forms of aggregation and embedding exist.
  • Empirical Validation: While their evaluation function shows promise, they suggest further validation through empirical analysis of human assessments of generated texts.
  • Broader Interactions: They propose extending their work to explore interactions between aggregation and referring expression generation, and testing interactions between or within other text generation modules.
  • Generalization: A larger-scale experiment is needed to generalize their claims about the importance of preferences.
  • Completeness of Features: The authors note that their set of features for evaluating embeddings is not claimed to be exhaustive.

Summary updated

BibTeX
@inproceedings{cheng2000experimenting,
  author = {Hua Cheng},
  title = {{Experimenting with the Interaction between Aggregation and Text Structuring}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2000},
}
Figure