Detecting Subtle Semantic Shifts with High Lexical Similarity

17 concepts9 papers1 proposal

Consider these two sentences:

  1. The medical board discussed performing the experimental surgery.
  2. The medical board performed the experimental surgery.

To a standard natural language processing model relying on word overlap or sentence embeddings, these sentences appear nearly identical. They share over 80% of their vocabulary, feature the same entities, and maintain a similar syntactic structure. Yet, the factual claim has completely flipped—an intention has been transformed into an action.

This tension defines the challenge of subtle semantic shift detection. Across applications ranging from summarization and translation to conversational AI, models frequently struggle when the surface-level vocabulary remains consistent but the underlying intent, truth value, or context drifts. Identifying these microscopic but critical alterations requires moving past lexical similarity toward deep semantic grounding.

The Illusion of Surface Similarity

Traditional evaluation metrics like BLEU or ROUGE operate primarily on text overlap. While effective at catching blatant hallucinations or completely unrelated text, they are blind to nuanced distortions.

As Yaghoub-Zadeh-Fard et al. observe in their study of crowdsourced training data, human workers often alter a single word or constraint when generating paraphrases. This minor lexical tweak fundamentally shifts the user's intent while preserving high surface similarity, creating a "dirty secret" of noisy data that off-the-shelf classifiers fail to catch.

This vulnerability extends to retrieval systems. In FAQ-based question answering, Xie et al. demonstrate that minor changes in query wording can signal an entirely different user intent. Without explicit grounding, retrieval models mistake these distinct intents for valid linguistic variations.

Furthermore, the problem is not just about changing words, but how words are framed. In Retrieval-Augmented Generation (RAG) for sensitive or controversial topics, Hoffmann et al. point out that models rarely invent facts out of whole cloth; instead, they introduce partial errors—such as shifting "consensus" to "unanimity." Because the model uses valid paraphrasing, simple word-overlap metrics fail to flag these distortions.

[Source Text] "The committee reached a consensus on the guidelines."
      │
      ├─► High Lexical Overlap / Valid Paraphrase (Passes standard metrics)
      │
[Summary]     "The committee was unanimous on the guidelines." 
              (X - Subtle Semantic Shift / Distortion)

Even advanced atomic fact-checking methods fall short here. As Hu et al. argue, breaking a summary down into isolated factual propositions can miss subtle shifts in tone, emphasis, or thematic focus that do not explicitly contradict the source text but still degrade the overall fidelity of the generation.

Disentangling Style from Intent

The core difficulty in detecting semantic drift lies in distinguishing between benign linguistic variations and malicious or erroneous meaning alterations.

In text editing and grammatical error correction (GEC), we expect the surface structure to change. However, reference-free metrics designed to evaluate these corrections often penalize valid improvements or, conversely, miss instances where a correction alters the core meaning. Sakai et al. highlight this gap, noting that evaluation tools require robust similarity estimators that can decouple grammatical fixes from undesirable semantic shifts.

A more extreme version of this problem occurs in journalism, where quotes are intentionally stripped of their surrounding context to alter a speaker's meaning—a practice known as contextomy.

[Original Context] "I would never support this bill unless small businesses are exempted."
                                      │
                                      ▼ (Contextomy)
[News Headline]    "Speaker declares: 'I would support this bill!'"

As Song et al. demonstrate, standard NLP tools easily identify the speakers and entities within a headline but fail to notice when editorial spin inverts the original quote's intent. The challenge is not detecting a change in topic—which out-of-distribution detection methods can occasionally flag when themes shift drastically within a language (DARRIN et al.)—but rather detecting when the vocabulary is identical but the logic is broken.

Mechanisms for Deep Semantic Grounding

To catch these elusive shifts, researchers are shifting away from raw text strings and moving toward structured, contrastive, and domain-specific representations that capture the underlying logic of an utterance.

1. Structured Meaning Maps (AMR)

Instead of treating text as a sequence of tokens, frameworks like FactGraph (Ribeiro et al.) convert both source documents and generated summaries into Abstract Meaning Representations (AMR). AMRs are directed graphs where nodes represent concepts (actions, entities) and edges represent semantic roles (who did what to whom).

By feeding both the raw text and these structured graphs into a dual-encoder system, the model can explicitly compare the topology of the claims. If a summary changes an edge from an intentional state ("discussed") to a completed state ("performed"), the graph structure changes dramatically, making the semantic shift visible even if the word-level similarity remains high.

2. Knowledge Anchors

In retrieval tasks, text can be grounded using external knowledge bases. The KAQA framework introduced by Xie et al. maps user queries and FAQ documents to explicit Knowledge Graph entities and semantic triples (e.g., [contactor, has_operation, recover]). By treating these triples as "knowledge anchors," the system bypasses surface-level phrasing and evaluates whether the core semantic relationship matches the target document.

3. Contrastive Learning with Hard Negatives

To train models to see the thin line between a benign edit and a semantic shift, researchers are leveraging contrastive learning. In the QuoteCSE framework developed by Song et al., models are trained using "hard negatives"—quotes taken from the exact same news article but representing a different context. This forces the model to learn the precise boundaries of intent preservation rather than relying on broad topical alignment.

4. Intermediate Labeling in High-Stakes Domains

In highly technical fields like medicine, semantic shifts can have severe consequences. For instance, confusing two different clinical interventions can invalidate an entire inference pipeline. Kim et al. address this by using language models to generate intermediate semantic labels before performing Natural Language Inference (NLI). Breaking down complex clinical trial reports into explicit intermediate assertions makes nuanced alterations in meaning visible to the downstream classifier.

By moving from surface-level text matching to structural, relational, and contrastive evaluation, these approaches allow machine learning systems to look past the words on the page and directly evaluate the preservation of intent.

Key Foundational Work

A Study of Incorrect Paraphrases in Crowdsourced User Utterances (2019) — contributed 3 concept elements to this direction.
FAQ-based Question Answering via Knowledge Anchors (2019) — contributed 1 concept elements to this direction.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations (2022) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_semantic_drift_detection_5bcf651c,
  title = {semantic drift detection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-drift-detection/rdr_5bcf651cbf584c629df6d2837de057b4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_semantic_drift_detection_5bcf651c_v1,
  title = {semantic drift detection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-drift-detection/rdr_5bcf651cbf584c629df6d2837de057b4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake