Abstract: This paper demonstrates how to significantly boost the accuracy of an older, rule-based symbolic parser (FRMG) using partially supervised learning on a modern treebank. By employing "imperfect oracles" to bridge the gap between incompatible annotation schemes, the authors tune the parser to achieve performance on par with modern statistical parsers while maintaining superior cross-domain robustness.
Introduction
In natural language processing, statistical parsers are typically trained and evaluated on the same treebank, ensuring that the parser's output perfectly matches the expected evaluation format. However, older "symbolic" or hybrid parsers—which rely on hand-crafted, linguistically motivated rules like Tree Adjoining Grammars (TAGs)—output structures native to their own underlying formalisms.
Comparing a symbolic parser to a statistical one is difficult because converting symbolic output to a treebank's format is often lossy. More importantly, training a symbolic parser on a modern treebank is challenging because there is no "reverse conversion" to translate gold-standard data back into the symbolic parser's native grammar. This work addresses how to use the French TreeBank (FTB) to train and improve the symbolic French parser FRMG despite these structural incompatibilities.
Methodology: Learning with Imperfect Oracles
To solve the incompatibility between the FTB training data and the FRMG parser, the authors employ partially supervised learning with imperfect oracles. Instead of mapping FTB trees directly into FRMG's internal rules, they parse training sentences using FRMG, convert the output to the FTB format, and compare it to the ground truth in a feedback loop.
- Reward: If the converted dependency matches the FTB gold standard, the algorithm assumes the parser made the correct internal choices and increases the weights of the heuristic rules that led to that choice.
- Penalize: If the dependency is incorrect, the rules that selected it are penalized. Because the system lacks a reverse-converter to identify the correct alternative, it distributes a positive weight update equally among all discarded competitor rules.
This offline process is iterated over the training data, gradually lowering a "temperature" parameter to fine-tune the weights of FRMG's edge-disambiguation rules.
Experimental Results
The primary dataset used is the French TreeBank (FTB), consisting of journalistic text converted into projective dependency trees. To test cross-domain robustness, the authors also utilized the Sequoia corpus (covering medical, encyclopedic, and political domains) and the EASyDev corpus.
Tuning FRMG via partially supervised learning improved its Labelled Attachment Score (LAS) on the FTB test set from a baseline of 82.08% to 87.17%. This 5-point jump brought FRMG roughly on par with contemporary statistical parsers such as the Berkeley parser (86.80%) and MALT (87.30%).
Discussion and Robustness
A key insight of this research is that symbolic parsers are highly robust to domain shifts. Purely statistical parsers tend to overfit to their training data (e.g., journalistic text). Because FRMG relies on a linguistically motivated grammar, it suffers a much smaller drop in accuracy when tested on out-of-domain text. On medical text, for instance, the statistical Berkeley parser lost 5.20 points of accuracy, while FRMG lost only 3.04 points.
Furthermore, the results suggest that noisy signals are sufficient for learning; a perfect 1-to-1 mapping between a treebank and a parser's internal grammar is not required to benefit from training data. Hand-crafted disambiguation rules, which can consult complex information about adjacent edges, continue to add significant value over standard machine learning features.
Conclusion
This study proves that symbolic parsers can be modernized through data-driven training without sacrificing their inherent linguistic strengths. While the conversion process and oracle strategies remain areas for future refinement, the hybrid approach achieves competitive accuracy and superior stability across diverse text styles.
Summary updated