Introduction
While modern statistical language models excel at local syntactic patterns, they often struggle with global semantic coherence. A model might correctly predict that "the" is followed by "cat," yet produce a sentence like "The cat sat on the blue idea"—grammatically plausible but semantically nonsensical. This research addresses the challenge of detecting whether a sentence "makes sense" by analyzing semantic relationships between word pairs across an entire sentence.
Quantifying Semantic Coherence
The core of the proposed method is the use of Yule’s Q Statistic to measure the strength of association between words. By calculating how often word pairs co-occur in a massive 104-million-word corpus of Broadcast News (BN), the researchers established a baseline for semantic correlation.
To distinguish real human-written text from "fake" sentences generated by a trigram model, the authors engineered approximately 70 features. Among the most powerful is the Likelihood Ratio, which models the probability of a sentence’s word-pair correlations appearing in a real corpus versus a synthetic one. This single feature achieved nearly 78% accuracy on its own.
The "Long-Range" Insight
A key finding of this study is that human-written sentences maintain semantic correlations even between words that are far apart. In contrast, trigram-generated sentences exhibit a sharp decline in correlation as the distance between words increases. This suggests that human coherence is a long-range phenomenon that local n-gram models fail to replicate.
Experimental Results and Human Benchmarking
The researchers trained a classifier using boosted decision trees (C5.0 and AdaBoost) on a dataset of 60,000 sentences (half real, half fake). The system achieved a classification accuracy of 80.37%, significantly outperforming previous baselines which reached only 74.30%.
To provide context, the researchers conducted a "Shannon-style" experiment with 30 human participants. Interestingly, the average human accuracy was only 72.65%. The automated classifier’s performance placed it among the top tier of human judges, demonstrating that statistical measures of semantic coherence can rival or exceed human intuition in identifying non-coherent text.
Conclusion and Future Directions
This work demonstrates that semantic coherence can be modeled effectively through word-pair statistics. By capturing long-range dependencies that local models miss, this approach offers a path toward more robust natural language processing. Future work involves integrating these coherence features into real-time systems, such as rescoring "n-best" lists in speech recognition to filter out semantically improbable transcriptions. This integration is estimated to reduce perplexity in standard language models by approximately 4.19%.
Summary updated