Jointly Parse and Fragment Ungrammatical Sentences

Homa B. Hashemi, Rebecca HwaAAAI 2018 · DOI

Introduction

Standard dependency parsers are typically optimized for well-formed text, aiming to produce a single, fully connected tree. However, when processing ungrammatical input—such as text from English-as-a-Second Language (ESL) learners—these parsers often "hallucinate" dependencies to satisfy structural requirements. These incorrect arcs create cascading errors in downstream tasks like semantic role labeling.

Comparison of syntactic dependency trees for ungrammatical and grammatical sentences. Red dashed lines indicate hallucinated arcs in the ungrammatical version.

The authors propose that for noisy text, it is more beneficial to produce a set of high-confidence parse fragments (accurate sub-trees) rather than a complete but erroneous tree. This paper introduces two methods to jointly parse and fragment sentences, pruning unreliable arcs in real-time.

Methodology

The paper moves away from traditional pipeline approaches (parse then prune) and introduces two "joint" models that learn to parse and prune simultaneously.

Joint Models

  1. RetrainedParser: Based on the transition-based SyntaxNet parser, this model was retrained on a specialized "ungrammatical treebank." In this setup, words associated with grammatical errors are assigned as the "root," effectively breaking the dependency arc and creating fragments.
  2. seq2seq (Sequence-to-Sequence): This model treats parsing and pruning as a translation task, mapping input sentences to a linearized dependency tree. Crucially, the authors introduced new "CUT" actions (@LCUT and @RCUT) into the shift-reduce vocabulary, allowing the model to explicitly prune dependencies it deems unreliable.

Training via Pseudo Gold Generation

To train these models, the authors developed a "Pseudo Gold" process. They aligned ungrammatical sentences with their corrected versions, parsed the corrected text, and projected those valid arcs back onto the original ungrammatical sentence. Arcs that could not be mapped due to errors were marked for pruning.

Experimental Results

The models were trained on approximately 576,000 sentences from major ESL corpora (FCE, NUCLE, and EF-CAMDAT) and evaluated on a 7,000-sentence subset of the FCE corpus.

Parsing and Pruning Performance

The seq2seq method significantly outperformed the RetrainedParser and previous pipeline baselines (H&H16) in both parsing accuracy (UAS) and the precision of its pruning decisions.

Performance comparison of different parsing methods. The seq2seq model achieves the highest UAS and F1 score for pruned arcs.

While the seq2seq model was conservative in its pruning, its decisions were highly reliable, achieving a precision of 0.71, compared to only 0.35 for the RetrainedParser.

Downstream Impact

The utility of parse fragments was demonstrated in Semantic Role Labeling (SRL). Using fragments generated by the seq2seq model nearly halved the False Discovery Rate (FDR) compared to a standard parser baseline (7.32 vs. 12.81). Additionally, fragment-based features proved effective for grammaticality judgment, achieving 81.3% accuracy in detecting whether a sentence contained multiple errors.

Conclusion and Future Work

The study demonstrates that joint parsing and fragmentation is a robust strategy for handling ungrammatical text. While the RetrainedParser faced limitations due to the rigid stack-based architecture of SyntaxNet, the seq2seq approach offered a flexible and accurate alternative. Future research may explore using these fragments to guide automated sentence correction rather than just error detection.

Summary updated

BibTeX
@inproceedings{hashemi2018jointly,
  author = {Homa B. Hashemi and Rebecca Hwa},
  title = {{Jointly Parse and Fragment Ungrammatical Sentences}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.11998},
}
Figure