Abstract
We present a robust parser which is trained on a treebank of ungrammatical sentences. The treebank is created automatically by modifying Penn treebank sentences so that they contain one or more syntactic errors. We evaluate an existing Penn-treebank-trained parser on the ungrammatical treebank to see how it reacts to noise in the form of grammatical errors. We re-train this parser on the training section of the ungrammatical treebank, leading to a significantly improved performance on the ungrammatical test sets. We show how a classifier can be used to prevent performance degradation on the original grammatical data.
Introduction
Most state-of-the-art natural language parsers are trained on the Penn Treebank, a collection of formal, edited Wall Street Journal (WSJ) articles. While these parsers achieve high accuracy on pristine text, they often lack "accurate robustness" when encountering real-world input. When a statistical parser meets a sentence containing a typo, a missing word, or a subject-verb agreement error, it typically attempts to force the ungrammatical input into a valid syntactic structure. This often results in a tree that misrepresents the speaker's intent, hindering downstream semantic analysis. This paper explores methods to adapt these parsers to handle such "noisy" grammar without sacrificing their performance on clean text.
Methodology
Synthetic Noise Generation
Due to the high cost of hand-labeling ungrammatical sentences, the authors developed a method to automatically inject synthetic noise into the Penn Treebank. They applied four primary operations—substitution, deletion, and insertion—to create common grammatical errors such as missing determiners, verb form mismatches, and extra words. Crucially, the underlying "gold standard" trees were modified minimally to ensure the parser's target remained the intended structure of the original sentence.
Parser Adaptation and Selection
The study utilized the Charniak and Johnson (2005) two-stage parser. Three primary training configurations were compared:
- $E0$ (Baseline): Trained exclusively on clean WSJ data.
- $E1error$: Trained on purely noisy data.
- $E1mixed$: Trained on a combination of clean and noisy data.
To prevent the adapted models from degrading performance on clean text, the researchers introduced a classifier strategy ($E2$). They tested an N-gram Classifier based on part-of-speech patterns and a Parse-Probability Classifier ($E2prob$). The latter runs each sentence through both the "clean" and "noisy" parsers and selects the output from the parser that reports a higher probability for its top-ranked tree.
Key Results
The evaluation demonstrated that standard parsers are highly sensitive to noise. The baseline $E0$ model saw its F-score drop from 90.9% on clean text to 82.1% on the "noisiest" development set. Retraining on mixed data ($E1mixed$) improved performance on noisy text to 85.7%, but at the cost of a slight performance dip on clean text (90.5%).
The $E2prob$ strategy emerged as the most effective, achieving the "best of both worlds." It maintained the original 90.9% F-score on clean WSJ text while significantly boosting performance on the noisiest test set to 85.4%. Interestingly, the parser's internal parse probability proved to be a more reliable indicator of noise than the external N-gram classifier.
Conclusion
This research demonstrates that synthetic noise generation is an effective proxy for training more robust parsers. By using a probability-based classifier to select between specialized models, it is possible to improve accuracy on messy, real-world text without compromising performance on formal English. Future work will focus on evaluating these techniques against naturally occurring human errors, such as those found in L2 learner corpora or social media text, to further validate the generalizability of synthetic training data.
Summary updated