Discourse Level Factors for Sentence Deletion in Text Simplification

Yang Zhong, Chao Jiang, Wei Xu, Junyi Jessy Li

TL;DR

This paper tackles the understudied problem of sentence deletion in text simplification, a crucial step for making complex documents accessible. By analyzing a novel, manually annotated dataset, the authors reveal that discourse-level factors, beyond just sentence content, significantly influence which sentences are removed. They then build predictive models that leverage these discourse features, achieving promising F1 scores for predicting sentence deletion.

What Problem Does This Paper Address?

Text simplification aims to make written content easier to understand for a wider audience, including children, non-native speakers, and individuals with language impairments. While much research has focused on simplifying individual sentences (lexical and syntactic changes), a significant operation in document simplification is sentence deletion. Simplified texts are often shorter than their originals, implying that entire sentences are removed. However, understanding why certain sentences are deleted and developing models to predict this deletion has been largely overlooked. This paper addresses this gap by investigating the factors that drive sentence deletion in document simplification, moving beyond sentence-level analysis to consider document- and discourse-level phenomena.

Methods

The authors employ a data-driven approach to analyze and predict sentence deletion.

  • Analysis: They conduct a detailed corpus analysis to identify factors influencing sentence deletion. This includes examining document characteristics (length, topic), rhetorical structure (using Rhetorical Structure Theory - RST), and discourse relations (inter-sentential relations and discourse connectives).
  • Prediction: To predict sentence deletion, they train classification models. They utilize a combination of sentence-level semantic features (GloVe embeddings) and document/discourse-level features. They experiment with logistic regression and feedforward neural networks. A key aspect is their use of "noisy supervision" from automatically aligned data for training, while evaluating on their meticulously manually annotated dataset. They also employ techniques like Gaussian feature vectorization for sparse features and downsampling to handle class imbalance.

Datasets

  • Newsela Corpus: The primary dataset used is the Newsela corpus, which contains multiple simplified versions of news articles written by professional editors for different readability levels (grades 3-12).
  • Manually Annotated Dataset: The authors introduce a novel, manually aligned dataset derived from 50 articles within the Newsela corpus. This dataset consists of 2,281 original sentences that have been carefully aligned to their simplified counterparts. Sentences in the original article that could not be mapped were considered deleted. This dataset serves as the ground truth for their analysis and evaluation.
  • Automatically Aligned Data: For training their predictive models, they leverage "noisy supervision" from 886 automatically aligned articles, totaling 42,264 sentences.

Key Insights and Ideas

  • Discourse Matters: The most significant insight is that sentence deletion is not solely determined by the content of an individual sentence. Discourse-level factors, such as how a sentence connects to its neighbors and its overall role in the document's structure, play a crucial role.
  • Editor Strategies Vary: Professional editors employ different strategies for simplification depending on the target reading level. Elementary school simplification involves more sentence deletion than middle school simplification.
  • Rhetorical Structure Salience: Sentences located deeper in the Rhetorical Structure Theory (RST) tree (indicating lower salience) are more likely to be deleted. Conversely, sentences that are nuclei of relations (more central) are less likely to be removed.
  • Connectives as Complexity Signals: The presence of explicit discourse connectives (like "however," "therefore") can signal sentence complexity, making them potentially more prone to deletion, especially in simpler reading levels. However, sentences with connectives at the beginning are more likely to be kept, suggesting editors are hesitant to remove sentences that establish clear discourse links early on.
  • Document Length Correlation: Longer documents tend to have a higher percentage of sentences deleted, suggesting a general tendency to shorten longer texts.

Key Results

  • Predictive Performance: Their best models achieved F1 scores of 65.2% for elementary school level simplification and 59.7% for middle school level simplification on their manually annotated test set.
  • Feature Importance: Document characteristics (like length and topic) were found to be more influential than sentence position for predicting deletion, especially at the middle school level. Discourse and rhetorical structure features, while insightful, did not significantly boost performance when added to models that already included document characteristics.
  • Manual vs. Automatic Alignment: The study highlights that automatic alignment methods can lead to significantly lower deletion rates compared to manual analysis, suggesting that automatic methods may over-align sentences.

Limitations and Future Work

  • Noisy Supervision: While using automatically aligned data for training is practical, it introduces noise that can affect model performance. The authors acknowledge that potential noise in partial matches from automatic alignment could render discourse relation signals less informative during training.
  • Feature Engineering: The paper notes that while rhetorical structure analysis provided interesting insights, encoding these as features did not further improve the predictive model. This suggests that capturing these complex relationships effectively for prediction remains challenging.
  • Future Directions: The authors hope their work will inspire further research into automatic document simplification. Future work could explore more sophisticated ways to incorporate discourse and rhetorical structure information, potentially through end-to-end neural models that learn these representations implicitly. Investigating the impact of different simplification strategies across various genres and languages could also be valuable.

Summary updated

Abstract

This paper presents a data-driven study focusing on analyzing and predicting sentence deletion -- a prevalent but understudied phenomenon in document simplification -- on a large English text simplification corpus. We inspect various document and discourse factors associated with sentence deletion, using a new manually annotated sentence alignment corpus we collected. We reveal that professional editors utilize different strategies to meet readability standards of elementary and middle schools. To predict whether a sentence will be deleted during simplification to a certain level, we harness automatically aligned data to train a classification model. Evaluated on our manually annotated data, our best models reached F1 scores of 65.2 and 59.7 for this task at the levels of elementary and middle school, respectively. We find that discourse level factors contribute to the challenging task of predicting sentence deletion for simplification.
BibTeX
@inproceedings{zhong2020discourse,
  author = {Yang Zhong and Chao Jiang and Wei Xu and Junyi Jessy Li},
  title = {{Discourse Level Factors for Sentence Deletion in Text Simplification}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i05.6520},
}
Figure