TL;DR
This paper presents a novel system for automatically condensing sentences into shorter, grammatically correct summaries using Lexical-Functional Grammar (LFG). By combining ambiguity packing, a transfer component, and a maximum-entropy model for disambiguation, the system achieves state-of-the-art summarization quality while guaranteeing grammatical output. They also propose a new parsing-based automatic evaluation method that correlates well with human judgments.
What Problem Does This Paper Address?
Traditional text summarization often relies on simply extracting and concatenating existing sentences. However, to create more concise and natural-sounding summaries, systems need to generate new sentences by condensing information. The core challenge here is ensuring these generated sentences are both grammatically sound and accurately capture the most important information from the original text. Existing approaches often struggle with guaranteeing grammaticality, especially when using statistical methods that might not inherently enforce linguistic rules. This paper aims to bridge this gap by leveraging the power of a linguistically rich grammar formalism (LFG) and advanced statistical techniques.
Methods
The system employs a multi-stage pipeline:
- LFG Parsing: An input sentence is parsed using a broad-coverage Lexical-Functional Grammar (LFG) for English. This produces a set of functional (f)-structures, which are represented in a "packed forest" format to efficiently handle ambiguities.
- Transfer Component: This component operates on the packed f-structures. It uses a set of rules to modify these structures, effectively deleting or altering parts of the sentence's meaning representation to create condensed versions. This component can also produce packed representations of potential condensations.
- Generator as Filter: The condensed f-structures are then passed through an LFG generator. This acts as a filter, ensuring that only structures that can be successfully converted back into grammatical English strings are kept. This guarantees the grammaticality of the final output.
- Stochastic Disambiguation: A maximum-entropy (MaxEnt) model is trained to select the "best" condensed f-structure from the set of grammatically valid options. This model is trained on pairs of original sentences and manually created gold-standard condensed f-structures. It uses "property-functions" that capture various linguistic features of the f-structures and the transfer rules used.
A key innovation is the use of ambiguity packing in the parser and transfer component, which allows for more efficient processing of multiple interpretations of a sentence.
Datasets
The authors used data from the experiments of Knight and Marcu (2000), which consists of sentence-condensation pairs extracted from computer-news articles and Ziff-Davis abstracts.
- Training Data: 436 sentence pairs were used for training the discriminative MaxEnt model.
- Test Data: Two test sets of 32 sentence pairs each were used. Testset I was taken from Knight and Marcu (2000), and Testset II was a random subset from the larger corpus.
- Gold Standard: For evaluation, they manually parsed and disambiguated 500 sentence pairs. This created a "gold standard" set of f-structures for both original and condensed sentences.
Key Insights and Ideas
- LFG for Guaranteed Grammaticality: By using LFG's constraint-based parsing and generation, the system inherently guarantees that its output will be grammatically well-formed, a significant advantage over purely statistical string-based methods.
- Parsing-Based Evaluation: The paper proposes and validates a novel automatic evaluation metric based on matching f-structures. This method is shown to correlate closely with human judgments, offering a more efficient and repeatable way to assess summarization quality than manual evaluation alone.
- Importance of Transfer Rules: The results suggest that learning what constitutes a good condensed f-structure (via the MaxEnt model) is more critical than having a perfectly disambiguated input f-structure for the transfer component.
- Ambiguity Packing Efficiency: The use of ambiguity packing in parsing and transfer is highlighted as a technique for efficiently handling the inherent ambiguities in natural language.
Key Results
- The proposed system achieves state-of-the-art summarization quality, with guaranteed grammaticality.
- The automatic parsing-based evaluation method shows a close correlation with manual evaluation, making it a reliable alternative for assessing summarization quality.
- Stochastic selection of transferred f-structures significantly reduces the error rate compared to random selection, even when starting with linguistically rich representations.
- The system's F-scores on the test sets were competitive, with the stochastic selection variant outperforming random selection and approaching the upper bound set by the best possible transfer output. For example, on Testset I, the system selection achieved an F-score of 67.3% compared to a lower bound of 58% and an upper bound of 77.2%.
Limitations and Future Work
- Packing in Generation/Stochastic Selection: While ambiguity packing is used in parsing and transfer, the current implementation requires unpacking for generation and stochastic selection. The authors note that extending packing techniques to these later stages is a direction for future work.
- Artificial Nature of Data: The authors acknowledge that the data used (from Knight and Marcu, 2000) might have an artificial nature, with human-written condensations often relying on simple deletion rather than more complex structural modifications (like nominalization or verb alternations). This limits the full exploitation of the system's generative capabilities.
- More Diverse Test Sets: To better evaluate the system's ability to handle a wider range of summarization strategies, they suggest creating more diverse test sets that include human-written condensations beyond simple deletions.
- Multiple Reference Condensations: For evaluation, they propose considering multiple reference condensations for each original sentence, similar to approaches in machine translation, to provide a more robust scoring mechanism.
Summary updated