Practical Issues in Automatic Documentation Generation

Kathleen R. McKeown, Karen Kukich, James ShawANLP 1994 · DOI

TL;DR

This paper introduces PLANDoc, a system that automatically generates 1-2 page summaries of telephone network planning activities. By combining a user-centered design approach with a constrained sublanguage, PLANDoc addresses the challenge of producing concise yet fluent documentation from complex, trace-based input, bridging the gap between raw system logs and human-readable reports.

What Problem Does This Paper Address?

In the telecommunications industry, planning engineers use complex software tools like Bellcore's LEIS-PLAN to create long-term capacity expansion plans for telephone networks. These plans are critical for managing infrastructure investments and ensuring service reliability. However, documenting the intricate decision-making process, including the exploration of various "what-if" scenarios and the rationale behind final choices, is a time-consuming and often manual task. This lack of automated, high-quality documentation poses several problems: it hinders knowledge transfer to new engineers, makes it difficult for management to understand and authorize expenditures, and can lead to costly legal issues if insufficient justification is provided. PLANDoc aims to solve this by automatically generating these crucial summaries from the system's trace logs.

Methods

PLANDoc's development was guided by a user-needs analysis, which revealed two key requirements: a user-centered design and a bounded sublanguage. The system architecture comprises five modules:

  • Message Generator: Translates raw PLAN system traces into a structured, attribute-value format.
  • Ontologizer: Enriches these messages with domain-specific semantic knowledge.
  • Content Planner: Determines which information to include, organizes it into paragraphs, and decides how to combine individual messages into complex sentences, managing discourse coherence and focus.
  • Lexicalizer: Maps semantic attributes to lexical roles and selects content words, using the FUF package.
  • Surface Generator: Takes the lexicalized messages and constructs linear surface sentences, handling syntactic roles, function words, and agreement, using the SURGE package.

A core innovation is the bounded sublanguage approach. Instead of allowing unlimited paraphrasing, PLANDoc restricts sentence generation to a set of four active and four passive sentence forms for most message types. This significantly reduces the combinatorial explosion of possible phrasings, making the system more manageable while still providing enough flexibility for fluency and conciseness. The system also systematically uses conjunction and ellipsis to combine related messages, avoiding repetition and creating more compact summaries.

Datasets

The paper describes the use of:

  • PLAN tracking files: These are raw logs of user interactions with the LEIS-PLAN system, serving as the input for PLANDoc.
  • Target narratives: Manually written summaries by an experienced retired planning engineer, based on PLAN runs. These were crucial for informing the structure, content, and vocabulary of the generated reports.

No new benchmarks were introduced; the focus was on generating practical documentation from existing system logs.

Key Insights and Ideas

  • User-centered design is paramount: Directly soliciting input from planning engineers and managers revealed critical needs, such as the ability to include manually written notes and select specific refinements for inclusion in the summary.
  • Bounded sublanguage is key to manageability: The sheer volume of potential paraphrases and sentence combinations from a rich domain like network planning is overwhelming. Constraining the system to a limited set of sentence structures (4 active, 4 passive) makes the problem tractable without sacrificing fluency.
  • Systematic conjunction and ellipsis are powerful tools: Combining related messages using conjunction and ellipsis is essential for generating concise summaries that avoid redundant information, a significant improvement over generating a separate sentence for each input fact.
  • Bridging the gap between raw logs and human-readable text: PLANDoc demonstrates a practical approach to transforming cryptic system traces into informative, human-understandable reports, a common challenge in many automated systems.

Key Results

The paper highlights the successful implementation of the tabular Input Summary generator and the textual Refinements Summary generator. The system is designed to produce 1-2 page summaries that integrate machine-generated text with engineer's notes. While specific quantitative results (e.g., accuracy metrics) are not detailed, the qualitative findings emphasize the system's ability to generate summaries that are both concise (avoiding repetition of information) and fluent (avoiding repetition of phrasing) due to the systematic use of conjunction and paraphrasing within a bounded sublanguage.

Limitations and Future Work

  • Limitations: The authors acknowledge that while the implemented algorithm is general, there are still cases that are exceptions to their approach, particularly when conjunction and paraphrasing interact in complex ways. Limiting paraphrasing has helped manage these exceptions.
  • Future Work:
    • The system is slated for full use in Fall 1994, allowing for a comprehensive evaluation of its output against user needs.
  • Augmenting the system to produce summaries of both the "base plan" and the "proposed plan." The proposed plan summary is more challenging due to tighter space constraints and the need to integrate information from multiple sources (base plan + selected refinements).
  • Further research is needed in discourse planning and selection of textual focus for generating more sophisticated summaries.

Summary updated

BibTeX
@inproceedings{mckeown1994practical,
  author = {Kathleen R. McKeown and Karen Kukich and James Shaw},
  title = {{Practical Issues in Automatic Documentation Generation}},
  booktitle = {ANLP},
  year = {1994},
  doi = {10.3115/974358.974361},
}
Figure