Accurate Function Parsing

Paola Merlo, Gabriele MusilloEMNLP 2005

TL;DR

This paper presents a method to integrate "function labels" (which describe the semantic or syntactic role of a phrase) directly into the parsing process, rather than as a separate post-processing step. By augmenting a state-of-the-art parser, they achieve top results in both parsing accuracy and function labeling, demonstrating that richer semantic information can be extracted without sacrificing syntactic parsing performance.

What Problem Does This Paper Address?

Modern natural language processing systems, like those used for question answering or flight booking, are moving towards understanding the meaning of text, not just its structure. To do this, systems need to go beyond basic syntactic parsing (identifying subjects, verbs, objects) and understand the function of different parts of a sentence. For example, understanding that "yesterday" in "I saw him yesterday" indicates time.

Traditional approaches often parse a sentence first to get its syntactic structure, and then a separate system tries to add these functional labels. This two-step process can be inefficient and may lead to errors. The challenge this paper addresses is whether it's possible to build a single, integrated system that can accurately perform both syntactic parsing and function labeling simultaneously, without degrading the performance of either task. This is particularly challenging because function labels are context-dependent and can sometimes conflict with purely syntactic roles, and because adding more complexity to the model requires more data to train effectively.

Methods

The core of this paper's approach is to augment an existing state-of-the-art statistical parser, the Simple Synchrony Network (SSN) parser. SSN parsers are chosen because they don't make strong independence assumptions and can adapt well to new tasks.

The key innovation is to train the SSN parser on Penn Treebank (PTB) data that has been re-labeled to include function labels. This means the parser learns to predict both the syntactic structure and the function of constituents (phrases or words) at the same time.

To improve performance, two augmentations were explored:

  1. Discriminating NULL labels: Initially, constituents without specific function labels are marked as "NULL." The paper hypothesizes that this "NULL" category is too broad and splits it into more specific types (like "OBJ" and "OTHER") to help the parser learn more precisely.
  2. Tag-splitting: This involves creating more fine-grained part-of-speech tags. For example, the tag "TO" might be split into "TO-DIR" and "TO-TMP" if it appears in a directional or temporal context, respectively. This helps the parser distinguish between different functional uses of the same word or tag.

Datasets

The primary dataset used is the Penn Treebank (PTB). The authors train their models on sections 2-21 of the PTB, validate on section 24, and test on section 23. The PTB is a widely used corpus of annotated English text, containing both syntactic structure and function labels. The paper notes that their work is related to other semantic role labeling resources like PropBank and FrameNet, but the direct focus is on the PTB's function labels.

Key Insights and Ideas

  • Integrated Parsing and Function Labeling is Feasible: The most significant insight is that it's possible to achieve state-of-the-art results in both syntactic parsing and function labeling within a single, integrated system. This challenges the notion that adding functional complexity inherently degrades parsing performance.
  • Function Labels are Informative: The success in function labeling suggests that current statistical parsing methods are general enough to capture shallow semantic information, which is crucial for deeper language understanding.
  • Tag-Splitting is More Effective than Just More Data: The paper found that creating more linguistically meaningful, fine-grained tags (tag-splitting) was more beneficial than simply increasing the vocabulary size by lowering the frequency cut-off for word-tag pairs. This highlights the importance of feature engineering and linguistic intuition.
  • NULL Labels are a Source of Error: The baseline model struggled with distinguishing between constituents that genuinely had no function label and those that were misclassified as NULL. Addressing this by refining NULL label distinctions improved performance.

Key Results

  • State-of-the-Art Performance: The augmented SSN parser achieved state-of-the-art results on both parsing (FLABEL-less scores) and function labeling (FLABEL scores) on the Penn Treebank.
  • No Degradation in Parsing: Crucially, the integrated system did not significantly degrade the performance on the standard syntactic parsing task compared to a parser trained without function labels. In some cases, performance even slightly improved.
  • Improved Function Labeling Accuracy: The augmented model showed significant improvements in function labeling accuracy, particularly for semantic labels. For example, the F-measure for semantic labels increased from 73.1% to 80.1% on the validation set.
  • Confusion Matrix Analysis: Table 2 (confusion matrix) revealed that the largest source of error for the baseline was confusion between specific labels and the "NULL" label, indicating the need for better discrimination.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the "function parsing task is more difficult than the simple parsing task." While their integrated approach mitigates performance degradation, the FLABEL scores are generally lower than FLABEL-less scores, indicating the added complexity.
  • The current work focuses on the Penn Treebank's specific function labels. Extending this to other annotation schemes like PropBank and FrameNet is a natural next step.
  • The paper mentions that the NULL label splitting was done into "CLR, OBJ and OTHER," with OBJ defined by specific syntactic conditions. Further refinement of these categories might yield further improvements.
  • Future Work:
    • The authors are currently working on extending their technique to PropBank annotation, which is a richer resource for semantic roles.
  • They suggest that the success of their left-corner parser, which relies on configurational criteria, in producing dependency output with function labels opens avenues for future exploration in dependency parsing.
  • Further investigation into how function labels describe dependency relations between heads and complements could lead to new insights for dependency parsing models.

Summary updated

BibTeX
@inproceedings{merlo2005accurate,
  author = {Paola Merlo and Gabriele Musillo},
  title = {{Accurate Function Parsing}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2005},
}
Figure