Sentence Complexity in Context

Benedetta Iavarone, Dominique Brunato, Felice Dell’OrlettaNAACL 2021

TL;DR

This paper investigates how the surrounding context influences human judgments of sentence complexity. They found that explicit linguistic features, especially when considering the full context, outperform powerful neural models like BERT in predicting perceived complexity, suggesting that current language models might not fully capture this nuanced aspect of language understanding.

What Problem Does This Paper Address?

Understanding sentence complexity is crucial for various NLP tasks like text simplification and readability assessment. While previous research has explored sentence complexity based on intrinsic linguistic properties (e.g., sentence length, syntactic structure), it has often treated sentences in isolation. However, human language comprehension models highlight the importance of context. This paper addresses the gap by systematically studying how the surrounding sentences (context) affect how humans perceive a sentence's difficulty, and whether this context can improve computational models' ability to predict this perception.

Methods

The authors collected a new dataset of English sentences, each rated for perceived complexity within three different contextual windows: "begin," "center," and "end." They then employed two main approaches to predict sentence complexity:

  1. Linear SVM with Explicit Linguistic Features: They extracted approximately 100 linguistic features covering raw text properties, morphosyntactic information, and syntactic structures. These features were used to train SVM regression models.
  2. BERT (Bidirectional Encoder Representations from Transformers): They fine-tuned a pre-trained BERT model to predict complexity.

Both approaches were evaluated using mean absolute error (MAE) on predicting both the average complexity score and the standard deviation of complexity judgments (indicating annotator agreement). They also analyzed correlations between specific linguistic features and complexity scores to understand which features are most influential.

Datasets

The study utilizes a newly collected dataset of approximately 2,900 English sentences. This dataset was built upon the crowdsourced data from Brunato et al. (2018), which contained 1,200 sentences. The key novelty of this new dataset is that sentences were presented and rated within three distinct contextual windows ("begin," "center," and "end"), allowing for the analysis of contextual influence.

Key Insights and Ideas

  • Context is King (for Humans): Humans demonstrably use surrounding sentences to judge a sentence's complexity. The perception of difficulty changes depending on whether a sentence appears at the beginning, middle, or end of a short passage.
  • Explicit Features Trump Black Boxes (for this task): Surprisingly, models relying on carefully engineered linguistic features (morphosyntactic and syntactic properties) outperformed state-of-the-art neural models like BERT in predicting perceived sentence complexity. This suggests that while BERT is powerful, it might not be effectively leveraging these specific features for this particular downstream task, especially with limited data.
  • Agreement Varies with Complexity: Annotators tend to agree more on simpler sentences than on more complex ones. The standard deviation of complexity ratings increases with the average complexity score, indicating less homogeneous perception for harder sentences.
  • Sentence Length Still Matters, but Context Amplifies: While sentence length remains a strong predictor of complexity, its influence is more pronounced and better exploited by models when context is considered.

Key Results

  • Predicting Average Complexity: SVM models using explicit linguistic features achieved better MAE than BERT models across all contextual windows. The best performance was observed when using features from the full context (all sentences in the window), especially for sentences of the same length.
  • Predicting Standard Deviation (Agreement): Contextual features were particularly crucial for predicting the standard deviation of complexity. SVM models showed significant improvement when using contextual information, suggesting that context helps explain why annotators agree or disagree on a sentence's difficulty.
  • Feature Importance: Features related to sentence length, numerical information, and syntactic structure (like parse tree depth and dependency link length) showed significant correlations with complexity judgments.

Limitations and Future Work

  • Data Size: The authors acknowledge that the dataset, while new, might be considered small for training complex neural models like BERT effectively. They suggest that more data could potentially improve BERT's performance.
  • BERT's Limitations: The findings suggest that current BERT models, despite their general linguistic capabilities, may not be optimally tuned to capture the nuances of perceived sentence complexity, especially when explicit linguistic features are available and the dataset is limited.
  • Future Work: The authors propose further investigation into how specific contextual windows impact complexity perception and how to better integrate contextual information into neural models. They also emphasize the importance of creating gold benchmarks that consider context for tasks like text simplification.

Summary updated

BibTeX
@inproceedings{iavarone2021sentence,
  author = {Benedetta Iavarone and Dominique Brunato and Felice Dell’Orletta},
  title = {{Sentence Complexity in Context}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure