TL;DR
This paper empirically evaluates the LFG-DOP (Lexical-Functional Grammar - Data-Oriented Parsing) model, a method for parsing sentences using fragments from a corpus of LFG-annotated examples. The authors introduce a novel probability model that treats generalized fragments as unseen events, leading to significantly improved parsing accuracy compared to the original LFG-DOP model.
What Problem Does This Paper Address?
Parsing sentences accurately is a fundamental challenge in Natural Language Processing. Traditional parsing models often struggle with the complexity and ambiguity inherent in human language. Data-Oriented Parsing (DOP) offers a data-driven approach, learning from a corpus of annotated examples to parse new sentences. However, applying DOP to richer linguistic representations like those from Lexical-Functional Grammar (LFG) presents challenges, particularly in how to probabilistically handle generalized fragments (fragments that represent more abstract linguistic structures). The original LFG-DOP model treated all fragments, generalized or not, with the same probability estimation method, which the authors suspected was suboptimal. This paper aims to empirically validate the LFG-DOP model and propose an improved probability estimation technique.
Methods
The core of the paper involves evaluating and extending the LFG-DOP parsing framework.
- LFG-DOP Model: The paper builds upon the LFG-DOP model, which uses LFG's c-structure (constituent structure) and f-structure (functional structure) representations. Sentences are parsed by decomposing LFG representations into fragments and then recomposing these fragments to analyze new sentences. The probability of a parse is determined by the probabilities of the fragments used in its derivation.
- Decomposition Operations: The paper details the decomposition operations (Root, Frontier, and Discard) used to create fragments from LFG representations. Discard is particularly important as it generates generalized fragments.
- Monte Carlo Parsing: A Monte Carlo technique is employed to estimate the most probable LFG-parse of a sentence. This involves sampling a large number of random derivations and selecting the most frequent one.
- Novel Probability Model: The key methodological contribution is a new probability model for fragments. Unlike Bod & Kaplan's original model, which treated all fragments equally, this new model distinguishes between "seen" fragments (generated by Root and Frontier) and "unseen" fragments (generated by Discard). It uses a discounting technique (specifically, the Turing-Good estimator) to assign a fixed probability mass to unseen fragments, preventing generalized fragments from absorbing too much probability mass and improving the estimation of seen fragments.
Datasets
The experiments were conducted on two LFG-annotated corpora:
- Verbmobil corpus: Contains appointment planning dialogues.
- Homecentre corpus: Contains Xerox printer documentation.
Both corpora were annotated at Xerox PARC and provided packed LFG-representations. The authors used a 90% training and 10% test split, with multiple random splits to ensure robustness.
Key Insights and Ideas
- Generalized Fragments are Tricky: The original LFG-DOP model's approach of treating generalized fragments (created by Discard) the same as specific fragments led to poor performance. This suggests that simply having more fragments doesn't automatically lead to better parsing if their probabilities aren't handled carefully.
- A Little Goes a Long Way (for Generalized Fragments): The proposed model, which assigns a small, fixed probability mass to generalized fragments, significantly boosts accuracy. This indicates that while generalized fragments are necessary for handling unseen linguistic phenomena, they don't need to dominate the probability distribution.
- LFG's Functional Structure Matters: Incorporating LFG's f-structures (functional information) into the parsing process, even when evaluating only c-structure accuracy, leads to a noticeable improvement over a pure phrase-structure (Tree-DOP) model. This highlights the value of richer linguistic annotations.
- Corpus Size is a Factor: The authors note that the contribution of generalized fragments to accuracy was statistically insignificant on their relatively small corpora. This suggests that for more robust estimation of generalized fragments, larger annotated datasets would be beneficial.
Key Results
- Improved Accuracy: The authors' modified LFG-DOP model significantly outperformed the original Bod & Kaplan model on both the Verbmobil and Homecentre corpora. For example, on the Verbmobil corpus, the exact match accuracy increased from 1.1% to 35.9% with the new model.
- Precision and Recall Boost: Precision and recall scores also saw substantial improvements. For instance, precision on Verbmobil rose from 13.8% to 77.5%.
- Functional Structure Benefit: When evaluating only c-structure accuracy, LFG-DOP (with functional structures) outperformed a Tree-DOP model (without functional structures) by a statistically significant margin. On the Verbmobil corpus, exact match accuracy for c-structures improved from 46.6% to 50.8%.
- Discard's Contribution is Minimal (on these corpora): Excluding Discard-generated fragments from the training set only slightly decreased accuracy in the authors' model, and these differences were not statistically significant.
Limitations and Future Work
- Corpus Size: The authors acknowledge that the small size of the corpora might limit the statistical significance of findings related to generalized fragments.
- Memory Constraints: Experiments were limited by memory, restricting the depth of indexed subtrees to 4.
- Future Work: The authors plan to test LFG-DOP using log-linear models, which are known to maximize the likelihood of the training corpus. They also suggest that their model could be adapted to exploit functional annotations in datasets like the Penn Treebank, provided they can be converted to LFG-style structures.
Summary updated