Learning Within-Sentence Semantic Coherence

Elena Eneva, Rose Hoberman, Lucian Vlad LitaEMNLP 2001

Introduction

While modern statistical language models excel at local syntactic patterns, they often struggle with global semantic coherence. A model might correctly predict that "the" is followed by "cat," yet produce a sentence like "The cat sat on the blue idea"—grammatically plausible but semantically nonsensical. This research addresses the challenge of detecting whether a sentence "makes sense" by analyzing semantic relationships between word pairs across an entire sentence.

Quantifying Semantic Coherence

The core of the proposed method is the use of Yule’s Q Statistic to measure the strength of association between words. By calculating how often word pairs co-occur in a massive 104-million-word corpus of Broadcast News (BN), the researchers established a baseline for semantic correlation.

To distinguish real human-written text from "fake" sentences generated by a trigram model, the authors engineered approximately 70 features. Among the most powerful is the Likelihood Ratio, which models the probability of a sentence’s word-pair correlations appearing in a real corpus versus a synthetic one. This single feature achieved nearly 78% accuracy on its own.

Figure 3: The distribution of the Likelihood Ratio feature shows a clear distinction between real Broadcast News and trigram-generated sentences.

The "Long-Range" Insight

A key finding of this study is that human-written sentences maintain semantic correlations even between words that are far apart. In contrast, trigram-generated sentences exhibit a sharp decline in correlation as the distance between words increases. This suggests that human coherence is a long-range phenomenon that local n-gram models fail to replicate.

Figure 5: At a distance of four words, the maximum Q value for real news sentences peaks significantly higher than for generated text, highlighting the persistence of semantic links in human writing.

Experimental Results and Human Benchmarking

The researchers trained a classifier using boosted decision trees (C5.0 and AdaBoost) on a dataset of 60,000 sentences (half real, half fake). The system achieved a classification accuracy of 80.37%, significantly outperforming previous baselines which reached only 74.30%.

To provide context, the researchers conducted a "Shannon-style" experiment with 30 human participants. Interestingly, the average human accuracy was only 72.65%. The automated classifier’s performance placed it among the top tier of human judges, demonstrating that statistical measures of semantic coherence can rival or exceed human intuition in identifying non-coherent text.

Conclusion and Future Directions

This work demonstrates that semantic coherence can be modeled effectively through word-pair statistics. By capturing long-range dependencies that local models miss, this approach offers a path toward more robust natural language processing. Future work involves integrating these coherence features into real-time systems, such as rescoring "n-best" lists in speech recognition to filter out semantically improbable transcriptions. This integration is estimated to reduce perplexity in standard language models by approximately 4.19%.

Summary updated

BibTeX
@inproceedings{eneva2001learning,
  author = {Elena Eneva and Rose Hoberman and Lucian Vlad Lita},
  title = {{Learning Within-Sentence Semantic Coherence}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2001},
}
Figure