Abstract
In a language generation system, a content planner embodies one or more "plans" that are usually hand-crafted, sometimes through manual analysis of target text. In this paper, we present a system that we developed to automatically learn elements of a plan and the ordering constraints among them. As training data, we use semantically annotated transcripts of domain experts performing the task our system is designed to mimic. Given the large degree of variation in the spoken language of the transcripts, we developed a novel algorithm to find parallels between transcripts based on techniques used in computational genomics. Our proposed methodology was evaluated two-fold: the learning and generalization capabilities were quantitatively evaluated using cross validation obtaining a level of accuracy of 89%. A qualitative evaluation is also provided.
Introduction
In Natural Language Generation (NLG), a "content planner" acts as the architect of a response, determining which information is relevant and how to sequence it to ensure the output is natural and logical. Traditionally, these planners rely on hand-crafted rules or "schemas" developed through intensive manual analysis by linguists and domain experts.
However, manual rule creation is labor-intensive and struggles with the inherent variability of spoken language. Domain experts, such as physicians, do not follow rigid templates when communicating. This variation makes it difficult for humans to identify consistent patterns and for traditional supervised learning methods to succeed without massive amounts of labeled data.
Methodology: A Genomics-Inspired Approach
The researchers propose a novel pipeline that treats transcripts not as strings of words, but as sequences of semantic tags (e.g., age, procedure, medication).
To identify patterns within these sequences, the system adapts techniques from computational genomics:
- Pattern Detection (Motif Discovery): Similar to how biologists identify recurring DNA sequences (motifs), the system uses an adaptation of the TEIRESIAS algorithm to find "semantic motifs." These patterns incorporate "don't care" positions (wildcards), acknowledging that while some elements have a fixed order, others are more flexible.
- Clustering: Initial detection often yields many similar patterns. The system employs agglomerative clustering to group these into broader "plan elements," which helps generalize the findings and filter out noise.
- Constraint Inference: The system calculates how frequently one cluster precedes another across the dataset. To ensure robust rules, it discards any constraints that are contradicted within the training data.
Data and Evaluation
The study utilized data from the MAGIC system, which generates briefings on patient status following coronary bypass surgery. The corpus consisted of 24 transcripts of residents briefing ICU nurses. These transcripts were semantically tagged using a set of 200 tags, eventually mapped to 29 broader categories. Despite the sequences being relatively short (averaging 33 tags), they exhibited high variability, representing a challenging "low-data" environment.
The methodology was evaluated using 3-fold cross-validation:
- Pattern Confidence: 84.62% (frequency with which learned patterns matched unseen test data).
- Constraint Accuracy: 89.45% (frequency with which predicted ordering held true in the test set).
Discussion and Conclusion
The most significant insight is that the "noise" in human speech—omitted details or reordered lists—is mathematically analogous to the mutations and gaps found in genetic sequences. By utilizing $<L, W>$ patterns (where $L$ elements must match within a window $W$), the system captures the structural "backbone" of a conversation while allowing for "floating" information.
Qualitatively, the machine-learned constraints were compared against a plan manually crafted by experts over three years. The automated system not only validated most existing rules but also discovered new placement constraints that human designers had previously overlooked. Future work will focus on scaling the system to larger datasets and incorporating communicative intention—the "why" behind specific ordering choices—which the current statistical model does not yet capture.
Summary updated