How do you correct run-on sentences it’s not as easy as it seems

Junchao Zheng, Courtney Napoles, Joel Tetreault, Kostiantyn OmelianchukEMNLP 2018

TL;DR

This paper tackles the challenging problem of automatically correcting run-on sentences, a common grammatical error often overlooked in existing research. The authors introduce two novel machine learning models, a CRF-based model (roCRF) and a Seq2Seq attention model (roS2S), which significantly outperform previous methods on this task. Crucially, they demonstrate that artificially generated training data can be effectively used to train these models, addressing the scarcity of annotated data for run-on sentences.

What Problem Does This Paper Address?

Run-on sentences, defined as two or more independent clauses joined without proper punctuation or conjunctions, are a persistent grammatical issue that hinders readability and can disrupt local discourse. While grammatical error correction (GEC) research has made strides in addressing common errors like preposition or article mistakes, run-on sentences have received surprisingly little attention. This is partly due to their nature as "sentence-level mistakes with long-distance dependencies," making them harder to correct than local errors. Furthermore, run-ons are relatively infrequent in existing annotated GEC corpora, making it difficult for systems to learn to correct them effectively. This paper aims to fill this gap by developing and evaluating models specifically designed for run-on sentence correction.

Methods

The authors propose two sequence labeling models to address run-on sentence correction:

  • roCRF (Conditional Random Fields): This model treats run-on correction as a sequence labeling task where each space between tokens is labeled as either needing a SPACE or a PERIOD. It leverages contextual features (tokens, POS tags, capitalization), parse features, and a language model perplexity measure to inform its decisions.
  • roS2S (Sequence to Sequence with Attention): This approach frames the problem as a neural sequence generation task. It employs a Seq2Seq model with an attention mechanism, similar to those used in neural machine translation. A bidirectional LSTM acts as the encoder, processing the input sentence, while a unidirectional LSTM serves as the decoder, generating the corrected output.

Datasets

The paper highlights a significant challenge: the scarcity of annotated run-on sentences in existing GEC corpora. To overcome this, they introduce:

  • FakeGiga-Train: Artificially generated training data created from clean newswire text (Annotated Gigaword). Run-on sentences are formed by removing terminal punctuation between adjacent sentences and lowercasing the start of the second sentence. This dataset comprises 2.8 million run-on sentences and 1.75 million negative examples.
  • FakeGiga and FakeESL: Test sets created using a similar artificial generation process, with FakeESL being a "noisy" version that may contain other grammatical errors.
  • RealESL: A test set derived from combining the NUCLE and CoNLL 2013/2014 Shared Task test sets, representing naturally occurring run-ons in learner English.
  • FakeESL-1%: A down-sampled version of FakeESL to match the low percentage of run-ons found in RealESL, enabling a fairer comparison.

Key Insights and Ideas

  • Artificial Data is Viable: The most significant insight is that artificially generated run-on sentences can serve as effective training data, performing nearly as well as models trained on naturally occurring errors. This opens up possibilities for addressing other low-frequency errors in GEC.
  • Run-ons are Different: The paper reinforces that run-on sentences are distinct from other GEC errors due to their long-distance dependencies, requiring specialized approaches.
  • Model Trade-offs: roCRF achieves higher precision, likely due to its use of syntactic features, while roS2S demonstrates superior recall and overall F0.5 score, indicating better generalization.
  • Existing GEC Models Struggle: Leading GEC models, even those trained on similar corpora, perform poorly on run-on sentences, suggesting a "reference bias" where models prioritize more frequent errors.

Key Results

  • Both roCRF and roS2S models outperform state-of-the-art baselines from punctuation restoration (Punctuator-EU, Punctuator-RO) and whole-sentence GEC (NUS18) on various test sets.
  • On the RealESL dataset (natural run-ons), roS2S achieves the highest F0.5 score of 0.67, while roCRF achieves very high precision (0.83-0.89) but lower recall.
  • The models trained on artificially generated data (FakeGiga-Train) generalize well to both artificial (FakeESL) and natural (RealESL) run-on sentences.
  • The NUS18 model, despite being trained on NUCLE (which RealESL encompasses), shows very poor performance on run-ons, highlighting the issue of low-coverage errors.

Limitations and Future Work

  • Inconclusive on Clean vs. Noisy Text: The paper notes that the results are inconclusive regarding how much harder the task is on noisy text compared to clean text.
  • Focus on Missing Periods: The current work primarily focuses on correcting run-ons by inserting periods. The authors acknowledge that other correction methods, such as adding conjunctions or converting clauses, are also valid but not addressed in this initial study.
  • Ungrammatical Text as Source: The authors suggest that using ungrammatical text as a source for artificial data is an area for future work.
  • Addressing Other Low-Coverage Errors: The paper strongly encourages future GEC research to address other infrequent but important errors that are often ignored due to reference bias.

Summary updated

BibTeX
@inproceedings{zheng2018how,
  author = {Junchao Zheng and Courtney Napoles and Joel Tetreault and Kostiantyn Omelianchuk},
  title = {{How do you correct run-on sentences it’s not as easy as it seems}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure