TL;DR
This paper investigates whether linguistic features related to text cohesion (how sentences and ideas connect) can accurately predict how difficult a text is to read, especially for second-language learners. They found that while these features are often assumed to be important, they don't significantly improve readability predictions beyond basic measures like sentence length and word frequency.
What Problem Does This Paper Address?
Readability formulas aim to estimate how easy a text is to understand. Traditional formulas (like Flesch-Kincaid) rely on simple metrics like sentence length and word complexity. However, researchers have long suspected that deeper linguistic properties, such as how well a text flows and connects ideas (cohesion and coherence), play a crucial role in comprehension. Previous studies have explored these "cohesive features" but yielded mixed results, with some showing promise and others failing to demonstrate significant predictive power. This paper aims to systematically evaluate a comprehensive set of 65 cohesion-based features, particularly for French as a second language (L2) learners, to clarify their actual contribution to readability assessment.
Methods
The researchers employed a multi-step methodology:
- Manual Annotation: They manually annotated a corpus of 83 French texts (from A2 to C1 CEFR levels, covering narrative and informative genres) for co-reference chains (multiple mentions of the same entity) and anaphoric chains (antecedent-anaphora pairs). This involved detailed annotation of mention types and syntactic functions.
- Automatic Annotation: They then used a rule-based tool called RefGen to automatically identify co-reference and anaphoric chains in the same corpus. This allowed them to compare the performance of manual vs. automatic annotation.
- Feature Extraction: They extracted 65 features categorized into:
- POS tag-based: Ratios of pronouns, articles, and proper names.
- Lexical cohesion: Measures of semantic similarity between sentences using LSA (Latent Semantic Analysis) and word overlap.
- Entity cohesion: Variables related to the transitions between syntactic functions of mentions in co-reference chains.
- Entity density: Measures of how frequently entities are mentioned.
- Co-reference chain properties: Proportions of different mention types (e.g., definite NPs, pronouns) within chains and their opening mentions.
- Classic features: Sentence length (NMP) and a unigram model for vocabulary difficulty, used as baselines.
- Correlation and Modeling: They performed Spearman correlations to assess individual feature effectiveness and semi-partial correlations (controlling for sentence length and word frequency) to isolate the unique contribution of cohesive features. Finally, they built readability models (SVM and SVR) using these features and compared their performance against baseline models.
Datasets
The study used a subset of the Corpus of FFL (French as a Foreign Language) texts gathered by François (2009). This corpus comprises 2,160 texts from 28 FFL textbooks, covering CEFR levels A1 to C2. For this study, they selected texts from levels A2 to C1, stratified by genre (narrative and informative), resulting in a corpus of 83 French texts. The corpus was annotated both manually and automatically for co-reference and anaphoric chains.
Key Insights and Ideas
- Cohesion's Overstated Role: The most striking finding is that despite their popularity, cohesive features don't significantly boost readability prediction beyond simple metrics like sentence length and word frequency. This challenges a long-held assumption in the field.
- The Power of Semi-Partial Correlation: Using semi-partial correlation proved crucial. It allowed the researchers to disentangle the true contribution of cohesive features from the influence of confounding factors like sentence length and word frequency, which are often correlated with cohesion.
- LSA's Nuance: While LSA-based lexical cohesion features showed initial promise, their predictive power vanished when controlling for sentence length and word frequency. This suggests that their apparent effectiveness might be an artifact of these simpler features.
- Manual vs. Automatic Annotation Matters: Significant discrepancies were observed between manually and automatically annotated features, particularly for complex co-reference chain properties. This highlights the challenges and potential pitfalls of relying solely on automated NLP tools for such nuanced linguistic analysis.
- Genre Differences Exist: The study confirmed that the relevance of certain cohesive features can vary between narrative and informative texts, suggesting that genre-specific models might be necessary for optimal readability prediction.
Key Results
- Individual Features: Only a handful of the 65 cohesive features showed significant correlation with readability after controlling for sentence length and word frequency.
- Readability Models: No model incorporating cohesive features significantly outperformed a baseline model based solely on sentence length and word frequency. The best performing models were SVM-based, but the improvements over the baseline were not statistically significant.
- Automatic Annotation Impact: Using automatically detected features sometimes led to slightly degraded performance compared to manual annotation, even when the automatic features showed better initial correlations.
- Genre-Specific Findings: For narrative texts, features like the number of chains and unique entities were relevant. For informative texts, LSA-based features and word overlap were more indicative of difficulty.
Limitations and Future Work
The authors acknowledge several limitations:
- L2 Corpus: The study was conducted on an L2 French corpus. The relevance of cohesive features might differ for native speakers (L1 texts).
- Language Specificity: While findings align with some English studies, results might vary across languages.
- Feature Set: Not all possible cohesive features were explored (e.g., mean distance between mentions within a chain).
- Annotation Tool: The RefGen tool has known limitations (e.g., handling complex referents, adverbs), which might have influenced results.
Future work could involve:
- Replicating the study on L1 corpora and other languages.
- Exploring a wider range of cohesive features and annotation tools.
- Investigating the interaction of cohesive features with other linguistic dimensions.
- Developing more robust automatic annotation methods for co-reference.
Summary updated