Abstract: Anne Vilnat, Gil Francopoulo, Olivier Hamon, Sylvain Loiseau, Patrick Paroubek, Eric Villemonte de la Clergerie. Coling 2008: Proceedings of the workshop on Cross-Framework and Cross-Domain Parser Evaluation.
2008.
The Bottleneck of Manual Annotation
In Natural Language Processing (NLP), large annotated corpora like treebanks are foundational for training statistical models and extracting grammars. However, manually annotating these resources is incredibly expensive, time-consuming, and prone to human error. For the French language, this has resulted in a limited volume of available training data.
The PASSAGE project addresses this bottleneck by exploring whether existing, imperfect parsing systems can be used to automatically annotate a massive corpus (10-100 million words). The challenge lies in standardizing the outputs of diverse parsers and combining them to create a "gold standard" resource that is more accurate than any single individual system.
The PASSAGE Methodology
The core of the PASSAGE approach is a feedback loop of ensemble parsing designed to reduce human effort while maximizing data quality. This methodology relies on three pillars:
- Standardized Annotation Format: To enable interoperability between different parsers, the team developed a unified XML-based format based on ISO specifications (MAF and SynAF). This format defines 6 types of syntactic constituents and 14 types of syntactic relations. By remaining parser-agnostic, it allows systems based on different linguistic theories to contribute to the same dataset.
- Web-Based Evaluation Service: A central web server allowed parser developers to submit their system's output and receive near real-time, confidential performance statistics against a gold standard. This immediate feedback loop proved essential for rapid system improvement.
- Parser Combination (ROVER): Inspired by speech recognition algorithms, the team uses a majority voting approach (ROVER) to combine outputs from 10 different parsing systems. When multiple architectures agree on a specific syntactic relation, the system assigns a high confidence score to that annotation, effectively creating high-quality data without human intervention.
Evaluation and Results
The project utilized the TECHNOLANGUE/EASY development corpus, consisting of 40,000 sentences, with a subset of 4,000 manually annotated sentences serving as the development gold standard. A separate, undisclosed set of 400 sentences was used for final testing.
Key findings from the evaluation phase include:
- High Accuracy in Constituents: Most participating systems achieved an F-measure between 80% and 90% for identifying syntactic constituents, with the top system reaching approximately 96.5%.
- Challenges in Relational Parsing: Identifying functional relations (e.g., subject-verb dependencies) proved significantly harder. As shown in the performance metrics for various participants, relational parsing scores generally remained below a 70% F-measure.
- Rapid Iteration Gains: The web evaluation server was a major success. During a one-month development phase, systems saw up to a 50% improvement in constituent parsing and a staggering 600% improvement in relational parsing from their initial baselines.
Conclusion and Future Directions
While the PASSAGE project has demonstrated the viability of automated large-scale annotation, challenges remain. Applying voting algorithms to unsegmented text requires complex alignment of tokenization and sentence boundaries across different parsers.
Moving forward, the project aims to finalize the combination strategy and use the generated annotations to enrich linguistic resources like lexicons. These enriched resources will then be fed back into the parsers, creating a continuous improvement loop for both the software and the resulting 100M+ word French treebank.
Summary updated