In modern natural language processing, we often assume that capturing complex human phenomena—like how difficult a text is to read—requires equally complex models. We build deep neural networks, parse intricate syntactic trees, and generate dense contextual embeddings. Yet, a persistent finding across decades of research challenges this assumption: simple, surface-level features like word counts, sentence lengths, and character frequencies are often the most robust and powerful predictors of text complexity.
Whether segmenting text into paragraphs (Sporleder et al. in Automatic Paragraph Identification: A Study across Languages and Domains) or evaluating text readability across diverse languages (Feng et al. in A Comparison of Features for Automatic Readability Assessment; Hancke et al. in Readability Classification for German using Lexical, Syntactic, and Morphological Features), basic structural properties consistently hold massive explanatory power. To understand why these shallow metrics perform so well, we must look at how they map to human cognition, how text aggregates over scale, and how they bypass the fragility of deep learning models.
Physical Length as a Proxy for Cognitive Load
The primary reason surface-level features work is that they serve as direct, unmediated proxies for human cognitive processing limits. When a person reads, their eyes move in rapid jumps (saccades) and pauses (fixations) to process text. Li et al. demonstrated in TorontoCL at CMCL 2021 Shared Task: RoBERTa with Multi-Stage Fine-Tuning for Eye-Tracking Prediction that basic properties like word length and word frequency capture a significant portion of the variance in human eye-movement patterns, often rivaling complex contextual models in data-constrained environments.
Because human working memory is constrained, longer sentences and longer words naturally increase cognitive burden. Consequently, tracking character frequencies, run lengths, and punctuation ratios allows systems to accurately score stylistic difficulty without any semantic comprehension. For instance, the authors of A Computer Readability Formula of Japanese Texts for Machine Scoring successfully built robust readability formulas for Japanese text using only these superficial characteristics.
This principle is so dominant that in text simplification tasks, reducing physical length outweighs deep structural reorganization. Lemke et al. found in Reference-less Quality Estimation of Text Simplification Systems that raw character and syllable counts are often more reliable predictors of sentence simplicity than complex linguistic formulas, confirming that reducing the physical footprint of text is the primary driver of perceived simplicity.
The Law of Large Numbers in Documents
While deep neural networks excel at capturing the nuanced meaning of an isolated phrase, their advantage often evaporates when evaluating longer texts. This phenomenon is driven by an aggregation effect: over the span of an entire document, micro-level semantic and stylistic variations tend to smooth out.
In You Don't Have Time to Read This: An Exploration of Document Reading Time Prediction, Weller et al. discovered that simple word count serves as a more effective predictor for document-level reading time than complex semantic or stylistic features. While deep learning models might better predict the reading time of an isolated, highly ambiguous sentence, a simple linear model using only the total number of words explains the vast majority of variance across full-length articles. Complex features offer minimal gains because document-level reading dynamics are fundamentally a function of volume.
Robustness in Low-Resource Environments
Deep learning models and advanced syntactic parsers require vast amounts of training data and highly specialized linguistic resources. When moving away from high-resource languages, these dependencies become liabilities. Surface-level features, by contrast, are universally available and highly transferable.
In a series of studies on educational materials for low-resource languages, traditional surface-level metrics consistently outperformed or matched complex alternatives:
- Filipino: Imperial et al. showed in Diverse Linguistic Features for Assessing Reading Difficulty of Educational Filipino Texts that traditional metrics like sentence length and word counts provide a foundational baseline that outperforms complex language-model features when used in isolation. Even when applying model-aware interpretation techniques to find the exact boundaries of reading difficulty, traditional features maintain high predictive power (Imperial et al. in Under the Microscope: Interpreting Readability Assessment Models for Filipino). When data is exceptionally scarce, classical machine learning models leveraging basic word counts, sentence lengths, and polysyllabic frequencies remain more informative than deep learning architectures (Imperial et al. in BasahaCorpus: An Expanded Linguistic Resource for Readability Assessment in Central Philippine Languages).
- Basque: Gonzalez-Dios et al. found in Simple or Complex? Assessing the readability of Basque Texts that simple lexical features alone are highly predictive of text complexity, outperforming models that incorporate a broader range of complex linguistic information.
This robustness even extends to cross-lingual semantic tasks. In evaluating cross-lingual textual entailment, Marchetti et al. found in Semeval-2012 Task 8: Cross-lingual Textual Entailment for Content Synchronization that the simple ratio of sentence lengths between two text fragments serves as a remarkably robust baseline for determining informational entailment, proving difficult for purely semantic models to beat.
Outperforming the Black Box
Perhaps the most striking evidence for the power of surface features is their ability to challenge or exceed the performance of state-of-the-art generative models and embeddings. Because surface features are highly interpretable, they avoid the "black box" vulnerabilities of deep learning, such as overfitting to domain shifts or breaking down when encountering grammatical errors.
In BATS: BenchmArking Text Simplicity , Kreutz et al. demonstrated that a rule-based approach using traditional linguistic features like sentence length and lexical richness can outperform complex embedding-based methods (such as OpenAI embeddings) in predicting human judgments of simplicity.
Furthermore, traditional readability formulas possess remarkable zero-shot capabilities. Lee et al. showed in Prompt-based Learning for Text Readability Assessment that traditional readability formulas serve as highly effective zero-shot baselines for pairwise difficulty comparison, often rivaling the performance of complex sequence-to-sequence neural models in unseen domains.
Even at the sentence level, simple lexical features—specifically the frequency and distribution of unigrams and bigrams—frequently outperform complex syntactic structures or higher-order n-grams, providing a robust signal even when text contains grammatical errors (Štajner et al. in Automatic Assessment of Absolute Sentence Complexity).
Ultimately, surface-level features dominate text complexity assessment because they capture the physical footprint of language. Human cognitive constraints are sensitive to this footprint—the number of characters we must decode, the number of words we must hold in working memory, and the familiarity of the vocabulary. By tracking these simple, length-based and frequency-based metrics, we capture the core constraints of human comprehension without needing to simulate full semantic understanding.
Key Foundational Work
A Computer Readability Formula of Japanese Texts for Machine Scoring (1988) — contributed 2 concept elements to this direction.
Automatic Paragraph Identification: A Study across Languages and Domains (2004) — contributed 1 concept elements to this direction.