Abstract
Shoaib Jameel, Xiaojun Qian, Wai Lam. Proceedings of COLING 2012.
2012.
Introduction
Traditional readability metrics, such as the Flesch-Kincaid Grade Level or the Automated Readability Index (ARI), primarily rely on "surface features" like syllable counts and sentence length. While these metrics are effective for general literature, they often fail in specialized technical domains such as Science or Psychology. For instance, the phrase "star network" consists of simple, one-syllable words, yet understanding its technical meaning requires domain-specific knowledge that surface-level formulas cannot capture.
Existing domain-specific readability methods typically require pre-defined ontologies or massive external lexicons, which are not always available for niche fields. This paper introduces an unsupervised framework for measuring document readability in specialized domains without requiring external knowledge bases. By modeling a document as a sequence of n-gram fragments in a latent semantic space, the authors calculate a "readability cost" based on term specificity and conceptual cohesion.
Methodology: The Sequential N-gram Connection Model (SNCM)
The core innovation of this research is the Sequential N-gram Connection Model (SNCM). This model treats the act of reading as a path through a document where the reader incurs a "cognitive cost" at each step. The framework operates through several key stages:
- Latent Semantic Indexing (LSI): All n-grams (unigrams, bigrams, and trigrams) and documents are mapped into a low-dimensional "concept space" using Singular Value Decomposition (SVD).
- Specificity: Within this LSI space, the proximity of an n-gram vector to the document vector determines its "specificity." High specificity indicates technical jargon or dense concepts, which increases the reading cost.
- Cohesion: This component measures the semantic relationship between consecutive fragments. Low cohesion between parts of a sentence indicates a disjointed text that is harder for a reader to process.
- Dynamic Programming: Rather than analyzing individual words in isolation, the model finds the optimal way to "fragment" a sentence into n-grams to minimize total cost, mimicking how a human reader groups words into meaningful chunks.
The authors developed two variants: SNCM1, which calculates specificity after the sequence is determined, and SNCM2, which integrates specificity directly into the fragment selection process.
Experimental Setup
To validate the model, the researchers crawled two large-scale web datasets:
- Psychology: 170,000 documents sourced from Wikipedia, Psychology Today, and similar outlets.
- Science: 300,000 documents sourced from ScienceDaily, ScienceForKids, and other educational platforms.
Ground truth was established by having undergraduate students annotate approximately 2,400 documents across both domains, rating them on a 5-point readability scale from "very low" to "very high."
Results and Discussion
The SNCM models were evaluated by re-ranking search results from an Information Retrieval (IR) engine (Okapi BM25) to prioritize easier-to-read documents.
- Superior Ranking Performance: Both SNCM1 and SNCM2 significantly outperformed traditional methods like ARI, Flesch, and SMOG. Using Normalized Discounted Cumulative Gain (NDCG@10), the SNCM models achieved scores up to 0.680 in Psychology and 0.713 in Science, whereas traditional formulas often hovered between 0.5 and 0.6.
- The Importance of Cohesion: A key finding was that cohesion—how well ideas link together—often mattered more for readability than the difficulty of individual words, particularly in the Psychology domain where vocabulary is more general but conceptual links are complex.
- Domain-Specific Nuances: In the Science domain, specificity played a much larger role in readability than in Psychology, likely due to the higher density of unique, technical jargon inherent to the hard sciences.
Conclusion
The SNCM framework demonstrates that unsupervised models can effectively capture domain-specific readability by focusing on conceptual density and flow rather than surface-level linguistic features. By using LSI, the model effectively builds its own internal understanding of a domain's jargon without needing a human-written dictionary. While computationally intensive, the model's adaptability makes it a powerful tool for specialized information retrieval. Future work may incorporate visual features such as layout, font size, and the presence of multimedia to further refine readability assessments.
Summary updated