Evaluating the Portability of Revision Rules for Incremental Summary Generation

Jacques RobinACL 1996 · DOI

Introduction

Natural Language Generation (NLG) systems often struggle to produce the dense, concise summaries characteristic of professional newswire writing. While simple facts can be listed easily, weaving secondary information into complex sentences requires sophisticated structural transformations. The STREAK system addresses this through an incremental revision model, starting with a basic draft and applying "revision rules" to polish the output. A key challenge in scaling such systems is the labor-intensive nature of rule creation. This study investigates whether revision rules developed for one quantitative domain (basketball sports reports) can be ported to another (stock market finance reports).

The Revision Rule Hierarchy

The STREAK model categorizes transformations into a hierarchy of rules. These are broadly divided into monotonic revisions, which add information without altering the existing draft's structure, and non-monotonic revisions, which involve deleting or rewording the draft to improve conciseness or flow.

The top-level hierarchy of revision rules, categorizing monotonic and non-monotonic transformations.

Monotonic rules include operations like Adjoin, Absorb, and Conjoin. Non-monotonic rules are more complex, encompassing Recast, Adjunctization, Nominalization, Demotion, and Promotion.

Methodology: Reverse Engineering Portability

To evaluate portability, the study employed a "reverse engineering" approach. Rules were first acquired from an 800,000-word corpus of NBA season reports. These rules were then searched for in a test corpus of over 18,000 stock market sentences using the CREP tool.

A rule was considered portable if its "signature"—a pair of sentences representing the trigger condition and the resulting transformation—could be identified in the new domain. Because domains differ lexically, search patterns were adjusted (e.g., mapping "team" to "market index") while maintaining the underlying syntactic structure.

Analysis of Rule Portability

The results indicate a high degree of structural generality across domains. Roughly 66% of the rules acquired in the sports domain were observable in the stock market domain. Specifically, 59% were fully portable (same-concept), while 7% were partially portable (different concept, same structure).

Detailed breakdown of Recast and Nominalization rules, illustrating the specific syntactic transformations found to be portable.

Interestingly, complex non-monotonic revisions like Recast (changing a classifier to a qualifier) and Nominalization were found to be highly portable. This suggests that the "logic of conciseness" is a general feature of professional reporting rather than a domain-specific quirk.

Discussion and Limitations

The primary friction in rule portability stems from thematic mismatches. For instance, sports rules often rely on antagonistic roles (Winner vs. Loser) that do not always have direct parallels in finance. The authors suggest that abstracting these thematic roles into broader categories (e.g., "Surface Objects") could increase portability to over 90%.

While promising, the study notes limitations: the evaluation required manual filtering of CREP results, and the analysis focused on structural rules rather than content selection or lexicalization. Future work should explore the real-world performance of a hybrid system and the separation of stylistic "side revisions" from core structural transformations.

Conclusion

This research demonstrates that nearly two-thirds of revision rules are portable across quantitative domains. This structural consistency suggests that building a general-purpose library of revision rules is a viable path toward more efficient and scalable incremental summary generation systems.

Summary updated

BibTeX
@inproceedings{robin1996evaluating,
  author = {Jacques Robin},
  title = {{Evaluating the Portability of Revision Rules for Incremental Summary Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {1996},
  doi = {10.3115/981863.981891},
}
Figure