A Production Rule System for Message Summarization

Elaine Marsh, Henry Hamburger, Ralph GrishmanAAAI 1984

Abstract

This paper details a 1984 research effort to automate the summarization of technical Navy equipment failure reports (CASREPs). By integrating natural language processing (NLP) with an expert system, the researchers developed a pipeline that transforms unstructured narrative text into a structured format, applies production rules to determine semantic importance, and selects key summary lines. The system achieved a 66.7% agreement rate with human experts, demonstrating the potential of hybrid AI systems in specialized technical domains.

Introduction

In the early 1980s, the U.S. Navy faced a data processing bottleneck with "Casualty Reports" (CASREPs). These reports documented equipment failures in technical narratives characterized by sentence fragments, jargon, and implicit causal links. While traditional expert systems were adept at diagnosis, they lacked the linguistic capabilities to process raw text. Conversely, standard NLP systems struggled to identify which specific facts within a narrative were most critical for decision-makers. This research addresses the gap by treating summarization as a process of both linguistic regularization and semantic inference.

Methodology

The system utilizes a multi-stage pipeline to bridge the gap between raw narrative and structured summary:

1. Natural Language Processing

Using the Linguistic String Project's parser, the system identifies grammatical relations and standardizes the text (e.g., converting passive voice to active). The output is mapped into a structured "information format" table. Each column represents a semantic category such as PART, STATUS, or CONNECTIVE, while each row represents a distinct clause or assertion.

Example of an information format table mapping technical components and their relationships.

2. The Production Rule System (OPS5)

Once formatted, the data is processed by 63 production rules (if-then statements). These rules perform two primary functions:

  • Inferencing: Identifying implicit relationships, such as determining that if one part "inhibits" another, the first part is the likely cause of the failure.
  • Scoring: Assigning numerical weights to rows. The system prioritizes "assembly-level" components—mid-level parts that are specific enough to be informative but broad enough to describe the overall failure.

Key Results

The system was evaluated against a test set of 12 casualty reports containing 38 sentences. The performance was measured by comparing the machine-selected summary lines against those chosen by Navy experts.

Comparison of machine-generated summaries vs. manual summaries across 12 test documents.

As shown in the results table, the system achieved agreement with human experts in 10 out of 15 instances (66.7%). At the message level, the system successfully identified at least one human-aligned summary line for 10 out of the 12 test messages. In cases where the system diverged from human selection, it often provided more granular causal details (e.g., identifying a specific bearing failure) which, while technically a "mismatch," provided high-value information.

Discussion and Limitations

A key insight of the study was the power of "sublanguages." By leveraging the restricted vocabulary of the Navy technical domain, the system could resolve ambiguities that would baffle a general-purpose NLP system. However, several hurdles remain:

  • Intersentential Processing: The system struggled with anaphora (e.g., resolving what "it" refers to across different sentences).
  • Knowledge Depth: Scaling the system would require massive, machine-readable equipment catalogs to understand complex part-whole hierarchies.
  • Selection Windows: Future iterations might benefit from selecting a "window" of high-scoring rows rather than a single top-ranked line to provide more contextually rich summaries.

Summary updated

BibTeX
@inproceedings{marsh1984production,
  author = {Elaine Marsh and Henry Hamburger and Ralph Grishman},
  title = {{A Production Rule System for Message Summarization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1984},
}
Figure