(Re)ranking Meets Morphosyntax: State-of-the-art Results from the SPMRL 2013 Shared Task

Anders Björkelund, Özlem Çetinoğlu, Richárd Farkas, Thomas Mueller, Wolfgang SeekerEMNLP 2013

TL;DR

This paper presents the IMS-SZEGED-CIS team's winning submission to the SPMRL 2013 Shared Task on parsing morphologically rich languages (MRLs). They achieved state-of-the-art results in both constituency and dependency parsing by combining strong baseline parsers with sophisticated reranking techniques, demonstrating the power of ensemble methods and high-quality preprocessing.

What Problem Does This Paper Address?

Morphologically rich languages (MRLs) pose significant challenges for natural language processing (NLP) tasks like parsing. Their complex morphology and often flexible word order lead to a vast lexical variation and out-of-vocabulary (OOV) issues, making it difficult for standard parsing algorithms to perform well. Developing effective parsers for MRLs typically requires language-specific expertise and tailored solutions, which is time-consuming. The SPMRL 2013 Shared Task aimed to address this by providing a common platform for evaluating parsing systems on a diverse set of MRLs, with varying levels of available preprocessing information (gold vs. predicted POS tags and morphological features).

Methods

The IMS-SZEGED-CIS team employed a two-pronged approach for the constituency and dependency parsing tracks, both leveraging a common preprocessing pipeline and a reranking strategy.

Preprocessing: * POS Tagging and Morphological Analysis: They used a multi-layered Conditional Random Field (CRF) tagger, MarMoT, to jointly predict POS tags and morphological information. This was crucial for handling the rich morphology of MRLs. * Morphological Dictionaries: They created language-specific morphological dictionaries by analyzing word forms and using various external tools (e.g., AraMorph, Apertium, HanNanum). These dictionaries were used as additional features for MarMoT. * Handling Unknown Words: For constituency parsing, they replaced rare words with their predicted morphological analysis, which proved more effective than standard signature-based methods.

Constituency Parsing: * PCFG-LA: They utilized products of Context Free Grammars with Latent Annotations (PCFG-LA) as their baseline parser. * Reranking: The output of the PCFG-LA parser was reranked to select the best parse tree.

Dependency Parsing: * N-best List Combination: They combined the N-best (top 50) outputs from multiple parsers (specifically, the mate parser and a best-first parser). * Reranking: A ranker was trained to select the best parse from the combined N-best list, using features derived from the individual parsers' scores and additional linguistic features. This approach was novel for dependency parsing.

Datasets

The paper evaluated their systems on datasets from the SPMRL 2013 Shared Task, which included a variety of morphologically rich languages. The specific languages are not explicitly listed as a single set but are mentioned throughout the text (e.g., Arabic, Basque, French, German, Hebrew, Hungarian, Korean, Polish, Swedish). The task involved different settings regarding the availability of gold or predicted POS tags and morphological information, as well as different training set sizes (full vs. 5k sentences).

Key Insights and Ideas

  • Reranking is Key: The paper strongly emphasizes that reranking, applied on top of strong baseline parsers (especially ensembles of parsers), is a highly effective strategy for achieving state-of-the-art results in parsing MRLs.
  • Preprocessing Matters Immensely: High-quality automatic POS tagging and morphological analysis are critical for the success of parsers on MRLs. The team's custom preprocessing pipeline significantly improved performance over the organizers' provided data.
  • Ensemble Power for Dependencies: Combining the N-best outputs of multiple parsers, each with different strengths and feature sets, provides a richer search space for the reranker, leading to better overall performance in dependency parsing. This is the first time this approach was successfully applied to dependency parsing.
  • Language Agnosticism with Flexibility: The core approach was designed to be language-agnostic, but the system allowed for fine-tuning by varying components, demonstrating a balance between generality and adaptability.
  • Handling Unknown Words: Replacing rare words with their predicted morphological tags was a simple yet effective way to mitigate OOV issues in constituency parsing.

Key Results

The IMS-SZEGED-CIS team achieved state-of-the-art results for all languages in both the constituency and dependency tracks, coming out first in both.

  • Constituency Parsing: Their system outperformed the organizers' best baseline for all languages in terms of PARSEVAL F1 scores.
  • Dependency Parsing: They achieved the highest scores for all languages except French, significantly outperforming their own baseline and other participants. For 5 languages, their baseline parser (with their own preprocessing) even outperformed the best other system.
  • Preprocessing Gains: Their custom preprocessing yielded improvements for most languages compared to using the organizers' provided preprocessing. For example, in Hebrew, their parser trained on their preprocessing was more than three points (LAS) ahead of parsers trained on the provided preprocessing.

Limitations and Future Work

  • French Dependency Parsing: While their system performed well overall, it did not achieve the top score for French in the dependency track.
  • Unary Chains in Polish: The paper notes that a specific issue with handling unary chains of length greater than 2 in the Berkeley parser negatively impacted Polish results. They suggest that collapsing unary chains could improve performance.
  • Feature Engineering for Reranking: While their general approach was language-agnostic, they found that language-specific features for the reranker yielded only minor improvements, suggesting that further research into effective feature engineering for MRLs is needed.
  • Combining Approaches: The authors suggest that their approach could potentially be combined with techniques from other teams or previous work on parsing MRLs to further improve parsing accuracy.

Summary updated

BibTeX
@inproceedings{bjorkelund2013re,
  author = {Anders Björkelund and Özlem Çetinoğlu and Richárd Farkas and Thomas Mueller and Wolfgang Seeker},
  title = {{(Re)ranking Meets Morphosyntax: State-of-the-art Results from the SPMRL 2013 Shared Task}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2013},
}
Figure