TL;DR
This paper evaluates and compares four French syntactic lexica by integrating them into a large-coverage parser (FRMG). The authors demonstrate that while switching lexica can lead to a slight performance decrease, error mining techniques can effectively identify and suggest improvements for these lexica, paving the way for higher-quality NLP resources.
What Problem Does This Paper Address?
Developing robust natural language processing (NLP) systems, particularly parsers, often relies on comprehensive syntactic lexica. However, many existing lexica, developed over decades, are not well-integrated into modern NLP pipelines, making their true utility for real-world applications difficult to assess. This paper addresses the challenge of evaluating and comparing these diverse syntactic lexica for French, aiming to understand their strengths, weaknesses, and potential for improvement when used within a state-of-the-art parser.
Methods
The core methodology involves converting four distinct French syntactic lexica (Lefff, LGLex, DICOVALENCE, and NewLefff) into a standardized Alexina format compatible with the FRMG parser. FRMG is a symbolic, large-coverage parser based on Tree Adjoining Grammar (TAG). The lexica are then plugged into FRMG, and their performance is evaluated on two different French corpora annotated with distinct schemes: the EASy/Passage format and the CoNLL dependency format.
A key technique employed is "error mining." This involves analyzing sentences that the parser fails to fully parse with a given lexicon. By comparing these failures against sentences that can be parsed (especially when using a reference lexicon like Lefff), the authors can pinpoint specific lexical entries or meanings that are likely causing parsing errors. This process helps identify missing information or incorrect encodings within the lexica.
Datasets
The paper uses two primary datasets for evaluation:
- EasyDev corpus: A corpus of approximately 4000 sentences, used in a previous EASy French parsing evaluation campaign. It is annotated with the EASy format, which includes chunks and dependencies.
- French TreeBank (CoNLL dependency format): A journalistic-style corpus converted to the CoNLL dependency format. This dataset is commonly used in international parsing evaluation campaigns.
The error mining process is performed on a larger corpus of 100,000 sentences (1.6 million words) named CPJ (Corpus Passage Jouet), which comprises various document styles (encyclopedic, news, discourse).
Key Insights and Ideas
- Lexica are Interchangeable (with caveats): The experiments show that it's feasible to swap different syntactic lexica within the FRMG parser, with only a relatively small decrease in performance. This highlights the modularity of the FRMG system.
- Error Mining is Crucial for Lexicon Improvement: Simply evaluating performance metrics isn't enough. Error mining provides a fine-grained, actionable way to identify specific lexical entries or meanings that are problematic, guiding targeted improvements.
- Granularity vs. Coverage Trade-off: Lexica with finer granularity (like LGLex and NewLefff) can sometimes lead to more precise parsing but may also increase parsing time and introduce new types of errors if not perfectly integrated.
- Existing Lexica Need Adaptation: Many older lexica, while rich in information, require significant effort to convert and adapt to modern NLP systems. This process often reveals missing information or incorrect encodings.
- The Importance of Lexical Representation: The paper underscores the need for lexica to capture various syntactic functions, argument realizations, and even clitic pronominalization, which FRMG's hypertag system can leverage.
Key Results
- Performance Comparison: While Lefff generally performs well, other lexica show varying results. DICOVALENCE, for instance, exhibits a noticeable decrease in full parse coverage on the EasyDev corpus. LGLex, despite its richness, shows increased parsing time.
- Error Identification: Error mining successfully identified specific issues in LGLex, such as missing clausal realizations for verbs like "mixer" and "zapper," and in NewLefff, where "estimer" lacked certain object realizations.
- Quantitative Performance: On the EasyDev corpus, Lefff achieved 83.45% coverage and 89.03% chunk F-measure. On the French TreeBank (CoNLL), Lefff achieved 89.53% coverage and 82.21% LAS (Labeled Attachment Score). These figures are generally competitive but slightly below state-of-the-art statistical parsers like MST.
Limitations and Future Work
- Preliminary Results: The authors emphasize that the CoNLL evaluation results are preliminary.
- Manual Effort Required: Error mining identifies suspects, but a significant manual effort is still needed to analyze all suspicious verbs and sentences to implement corrections.
- Adaptation Effort: Integrating and optimizing lexica within FRMG requires considerable effort, including adding new features and correcting existing entries.
- Future Work: The authors plan to continue applying error mining to larger corpora and to all lexica. They also suggest exploring partially supervised learning techniques to further boost FRMG's disambiguation performance, potentially reaching levels comparable to statistical parsers. The development of better-quality, freely available lexica for NLP systems is a key long-term goal.
Summary updated