TL;DR
This paper tackles the problem of incomplete coverage in FrameNet, a crucial resource for frame semantic parsing. They propose and evaluate a method to automatically expand FrameNet's training data by leveraging the Paraphrase Database (PPDB), achieving a significant increase in coverage while maintaining reasonable precision.
What Problem Does This Paper Address?
FrameNet is a valuable lexico-semantic dataset that captures the meaning of words in relation to real-world events and entities. It's the backbone for frame semantic parsers, systems that analyze sentences to identify evoked frames and their associated elements. However, like many manually constructed resources, FrameNet suffers from incomplete coverage. This means that many words or concepts that a parser might encounter in real-world text are "out-of-vocabulary" (OOVs) for FrameNet. Current parsers often rely on heuristics to handle these OOVs, but this paper proposes a more direct solution: augmenting FrameNet itself. By expanding the training data, the goal is to enable existing parsers to retrain on a richer dataset, eliminating the need for complex OOV resolution at inference time.
Methods
The core of this paper's approach is to use the Paraphrase Database (PPDB) to expand FrameNet. PPDB is a large collection of lexical, phrasal, and syntactic paraphrases. The authors focus on using PPDB's lexical rules, which identify word-to-word paraphrases.
Their method involves two main experiments:
- Direct Paraphrasing: For each target word in a FrameNet annotated sentence, they query PPDB to find potential paraphrases.
- Enlarged Lexical Units: They first expand the set of lexical units associated with a frame by considering frames related through specific frame-frame relations (like "Is-Perspectivized_In" or "Is_Inherited_By"). Then, they query PPDB for paraphrases of all these expanded lexical units. This simulates a scenario where a frame is mapped to a broader set of words.
They use a language model (KenLM) to score the generated paraphrases for grammatical correctness and fluency. They also employ WordNet to filter out paraphrases that are simply synonyms, aiming for more novel expansions.
Datasets
- FrameNet: Used as the primary resource to be augmented. The paper specifically mentions using fulltext annotated sentences.
- PPDB (The Paraphrase Database): Used as the source of paraphrases to expand FrameNet. They experiment with different sizes of PPDB (S, M, L, XL).
- WordNet: Used for filtering out simple synonyms to focus on more meaningful expansions.
- Annotated Gigaword: Used to train the language model (KenLM) for scoring paraphrases.
Key Insights and Ideas
- PPDB as a powerful augmentation tool: The paper demonstrates that PPDB can effectively generate paraphrases that can be used to expand FrameNet's coverage.
- The importance of frame-frame relations: Experiment 2, which leverages frame-frame relations to expand the set of target words for paraphrasing, shows a significant increase in the number of potential paraphrases. This highlights the interconnectedness of frames and how understanding these relationships can lead to richer expansions.
- Quality vs. Quantity Trade-off: While larger PPDB sizes (like XL) generate more paraphrases, the quality (grammaticality and correct frame invocation) can vary. Filtering with language models and WordNet is crucial for obtaining useful data.
- Limitations of current FrameNet: The paper implicitly highlights the sparsity of FrameNet by showing how many new examples can be generated, and also points out instances where FrameNet might lack frames for common concepts (e.g., "Programming").
Key Results
- Coverage Increase: The paper reports a "three-fold increase in coverage at 65% precision" when augmenting FrameNet with PPDB.
- Quality of Paraphrases: For a sample of 25 target words, the top three paraphrases produced by PPDB XL evoked the correct frame and were grammatical 65% of the time.
- Impact of PPDB Size: Table 1 shows that larger PPDB sizes (L, XL) significantly increase the average count of lexical units per frame compared to smaller sizes.
- Filtering Effectiveness: Table 3 indicates that filtering with WordNet (columns 1b and 2b) reduces the number of generated paraphrases but likely improves their quality by removing simple synonyms.
Limitations and Future Work
- Ungrammaticality and Inaccuracy: The authors acknowledge that some generated paraphrases are ungrammatical or inaccurate. They identify two main reasons:
- Multi-word Expressions: FrameNet sometimes annotates only a single token as a trigger when it's part of a larger expression.
- Inaccuracy in PPDB: For some words, PPDB itself might provide inaccurate paraphrases, especially when it has a limited number of candidates.
- Contextual Application: A significant challenge remains in recognizing the specific contexts in which a generated paraphrase is appropriately applied.
- Future Directions:
- Increase the precision of paraphrases and improve the ability to recognize their appropriate contexts.
- Augment other resources like PropBank or the ACE corpus using similar methods.
- Improve PPDB's accuracy, particularly for larger settings.
- Use better language models (e.g., recurrent neural networks) for scoring paraphrases.
- Utilize crowdsourcing platforms like Amazon Mechanical Turk for more robust evaluation of paraphrase quality.
Summary updated