Semantic Coherence as a Confidence Measure for Noisy Text

21 concepts12 papers1 proposal

When automatic speech recognition (ASR) systems, optical character recognition (OCR) engines, or language learners generate text, they frequently make errors that are locally plausible but globally bizarre. An acoustic model might mishear a word as a phonetically similar counterpart, or an OCR engine might misread a ambiguous cursive character. Because these primary systems rely heavily on local signals—such as acoustic frames or localized character n-grams—they suffer from a blind spot: they cannot easily perceive when a chosen word completely derails the broader topic of conversation.

To resolve these local ambiguities, a parallel line of research introduces a global "sanity check" based on semantic coherence. The core intuition is that natural human language is topically consistent; words within a sentence or document cluster within a shared semantic space. By measuring how much an individual word's semantic profile aligns with the aggregate profile of its context, computational systems can flag "semantic outliers" as highly probable errors.

Quantifying Semantic Disruption

To transform this intuition into a functional error-detection mechanism, systems must mathematically quantify how well a word fits its environment.

One prominent approach leverages topic modeling frameworks like Latent Dirichlet Allocation (LDA). As Chen et al. demonstrate, a system can extract the latent topic distribution of an entire utterance and compare it directly to the topic distribution of each individual candidate word. If a word's topic distribution diverges sharply from the aggregate context, it indicates an out-of-topic recognition error that traditional language models often miss.

[ Local Acoustic/Visual Signal ] ---> Generates Candidate Word
                                              |
                                              v
[ Global Context Window        ] ---> Extracts Latent Topic Vector
                                              |
                                              v
                                   Is the Word an Outlier?
                                   /                     \
                             [ Yes ]                     [ No ]
                                /                           \
                    Flag as Potential Error          Accept Word

Rather than just comparing static vectors, systems can also measure the dynamic disruption caused by a word. Herbelot et al. calculate the overall topic coherence of a sentence context both with and without a target word. If removing the word significantly improves the surrounding text's coherence, the word is flagged as a semantic anomaly.

Alternatively, instead of global topic models, systems can evaluate sliding windows of local semantic similarity. Inkpen et al. compute the statistical similarity of a word relative to its immediate neighbors. If a word's individual coherence score falls below a defined threshold relative to the average coherence of its neighbors, it is isolated as a mistranscription. This computational strategy mirrors the classic "word intrusion" task analyzed by Baldwin et al., where statistical association features are used to automatically isolate an artificial intruder word hidden inside an otherwise coherent group of terms.

The Asymmetry of Coherence

A foundational insight in this domain is that semantic coherence operates asymmetrically. In his work on speech recognition confidence measures, Cox discovered that semantic similarity metrics are unexpectedly more effective at identifying correctly recognized words than incorrect ones.

Why does this happen? A recognition error might occasionally yield a word that happens to be semantically neutral or tangentially related to the topic by pure chance. However, the inverse is rarely accidental: if a word aligns perfectly with a niche, multi-word semantic theme, the probability that it was generated by random acoustic or visual noise is exceptionally low. Semantic coherence is therefore a stronger indicator of accuracy than semantic incoherence is of an error.

This asymmetry makes semantic validation highly effective at reducing false positives in error-detection pipelines. For instance, in Out-of-Vocabulary (OOV) detection, an ASR system flags words it does not recognize as potential errors. To prevent the system from misidentifying rare but correctly spoken words as errors, Linarès et al. and Lecouteux et al. introduce a post-classification semantic filtering stage. By leveraging Latent Semantic Analysis (LSA) and external web-based co-occurrence data, they check whether the suspected OOV word fits the global theme of the discourse. If it aligns semantically, the error flag is dismissed, significantly boosting the precision of the detection system.

A Universal Sanity Check

Because semantic consistency relies on global discourse properties rather than specific acoustic or visual modalities, it serves as a highly transferable validation layer across diverse domains:

  • Spoken Document Retrieval: Senay et al. show that identifying and filtering out semantic outliers from automatic transcriptions prevents noisy, misrecognized words from corrupting downstream document indexes, ensuring high-quality information retrieval even when individual word confidence scores are ambiguous.
  • Optical Character Recognition: For cursive or morphologically complex scripts like Arabic, Tomeh et al. use semantic coherence features derived from massive external corpora to rerank the top hypotheses of an OCR engine, correcting character-level errors by choosing the candidate that is most semantically plausible.
  • Clinical and Structural Validation: This framework extends beyond machine errors to human language variations and data curation. Rouhizadeh et al. apply distributional semantic models to automatically identify "unexpected" words in the speech of children with autism, using external lexical resources like WordNet to verify whether a word is a true semantic outlier or merely a valid synonym. Even in structured text processing, Bögel et al. demonstrate that enforcing consistency between word-sense disambiguation and coreference resolution can automatically flag errors in complex digital humanities annotations, operating on the principle that mentions belonging to the same discourse entity must maintain semantic coherence.

Ultimately, as demonstrated early on by Eneva et al., global word-association features can allow computational systems to match or exceed average human performance in identifying nonsensical sentences. By treating text generation not just as a sequence of characters or sounds, but as a continuous web of topical relationships, semantic coherence provides a robust, high-level defense against local processing errors.

Key Foundational Work

A semantically-based confidence measure for speech recognition (2000) — contributed 1 concept elements to this direction.
Learning Within-Sentence Semantic Coherence (2001) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_semantic_topic_consistency_c4a004a2,
  title = {semantic topic consistency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-topic-consistency/rdr_c4a004a2c806454cbb6435f17565e5ee},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_semantic_topic_consistency_c4a004a2_v1,
  title = {semantic topic consistency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-topic-consistency/rdr_c4a004a2c806454cbb6435f17565e5ee/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake