In modern natural language processing, contrastive learning has become a standard tool for teaching models to distinguish between related and unrelated information. Typically, this involves training a model to pull a "positive" pair (like a query and its answer) closer together in vector space while pushing "negative" pairs (the query and a random sentence) further apart.
However, a significant problem arises when these negative examples are sampled randomly from a large corpus. If a model is asked to find an answer about "Mars exploration" and the negative example is a sentence about "baking bread," the task is trivial. The model learns to distinguish between broad topics rather than understanding the nuances of the query. As Zhang et al. argue, models trained on these "easy negatives" often fail to capture the fine-grained semantic distinctions necessary to differentiate between closely related segments within the same document.
To solve this, researchers have turned to intra-document negative sampling: the practice of selecting negative training examples from the same local context as the positive pair.
Breaking the Topic Shortcut
The intuition behind intra-document sampling is to eliminate "shortcuts." When a negative example comes from the same document or paragraph as the positive example, they share the same vocabulary, style, and overarching topic. This forces the model to move beyond keyword matching.
For instance, in Answer Sentence Selection (AS2), Garg et al. and Di Liello et al. demonstrate that selecting non-answer sentences from the same paragraph as the correct answer improves discriminative power. Because these sentences are topically relevant but factually insufficient, the model must learn to identify the specific semantic cues that satisfy a query. Similarly, Hong et al. propose treating the exact sentence containing an answer as the positive sample while using all other sentences in that same passage as "hard negatives." This localized pressure reduces false positives by teaching the model to ignore topically related but irrelevant context.
Learning Discourse and Coherence
Beyond simple retrieval, intra-document sampling is essential for understanding how language flows. If a model only sees sentences from different documents, it never learns the "connective tissue" of a narrative.
Iter et al. introduced CONPONO, which trains models to distinguish a specific neighboring sentence from a "lineup" of decoys taken from the same local context. By forcing the model to pick the correct next sentence among other sentences from the same story, the model develops a superior understanding of discourse. Xu et al. apply a similar logic to text coherence, replacing a sentence with another from the same document to force the model to recognize subtle logical transitions rather than broad thematic shifts.
Resolving Ambiguity in Complex Documents
In specialized domains like law or news, different sections of the same document often use nearly identical technical language. Standard retrievers struggle here because the vector embeddings for two different paragraphs in a legal brief might look almost identical.
The authors of QBR address this by using contrastive learning to increase the vector distance between different segments of the same document. This "fine-grained" separation allows a system to pinpoint the exact paragraph that answers a user's question, even when surrounding paragraphs share the same vocabulary. In the realm of misinformation, Song et al. use other quotes from the same news article as hard negatives to detect "contextomy"—the act of taking a quote out of context to change its meaning. By training on these intra-document negatives, the model becomes sensitive to the subtle shifts in intent that occur when a statement is moved within a single source.
Improving Faithfulness and Grounding
Intra-document sampling also provides a path toward more reliable AI assistants. A common failure in Retrieval-Augmented Generation (RAG) is "hallucination," where a model provides a fluent answer that isn't actually supported by the source text.
Naseem et al. address this by generating synthetic negative examples: they swap answers between different queries within the same document. This creates a training pair that is linguistically fluent and contextually related but factually incorrect for that specific prompt. This forces the model to prioritize precise "grounding" in the text over general topical relevance.
By shifting the focus from what a document is about to how its specific parts differ, intra-document negative sampling transforms language models from broad topic-matchers into precise semantic discriminators.
Key Foundational Work
TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selection (2020) — contributed 1 concept elements to this direction.
Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models (2020) — contributed 1 concept elements to this direction.
A Cross-Domain Transferable Neural Coherence Model (2019) — contributed 1 concept elements to this direction.