Semantic Similarity for Automated Pseudo Labeling

37 concepts29 papers1 proposal

Supervised machine learning models excel when provided with dense, granular labels—such as knowing exactly which sentence in a document contains a key fact, or which specific word maps to a region in an image. However, collecting these fine-grained annotations creates a severe human bottleneck, as demonstrated by Yllias Chali et al..

To bypass this bottleneck, a powerful paradigm has emerged: automated reference-based pseudo-labeling. The core intuition is that while granular labels are expensive, "macro-level" pairs—such as a long document paired with a human-written summary, or a conversation history paired with a final response—are often readily available. Because a summary or response inherently distills or paraphrases the most critical source information, researchers can reverse-engineer granular training signals. By measuring the semantic or lexical similarity between individual source components (sentences, words, or facts) and the global reference, models can automatically generate "silver" or "pseudo" labels to guide fine-grained learning objectives without manual intervention.

From Lexical Overlap to Dense Semantic Alignment

The earliest iterations of this technique relied on surface-level lexical overlap or sequence alignment metrics to establish relevance. In complex question answering, Yllias Chali et al. calculated the token overlap between source sentences and human-authored reference summaries to algorithmically label training data. Similarly, Gerasimos Lampouras et al. utilized the ROUGE-W sequence similarity metric to compare candidate web snippets against existing encyclopedia entries, successfully training specialized definition extraction tools. In document summarization, Wei Li et al. derived silver importance labels by computing the lexical similarity between source sentences and ground-truth summaries, using these scores to train an internal information selection layer.

However, exact lexical matching is inherently fragile; it fails when a reference summary or response uses synonyms or paraphrases instead of direct quotes. To overcome this, modern architectures leverage dense embeddings and semantic similarity metrics (such as cosine distance or BERTScore) to capture deeper conceptual alignment.

For example, in open-retrieval conversational question answering, Chen Qu et al. demonstrated that shifting from rigid string-matching to semantic similarity drastically increases the volume of usable training data. This shift allows models to successfully learn from natural, freeform human paraphrases that traditional extractive methods would otherwise ignore due to lexical mismatches.

Applications in Summarization and Information Selection

Using a global summary as an oracle to label fine-grained source text has become a standard mechanism for training extractive and hierarchical models. In narrative analysis, Jessica Ouyang et al. identified the "most reportable event" or turning point of a story by measuring the semantic similarity between individual sentences and a global summary of the text, leveraging the summary as a proxy for structural importance.

This approach is particularly vital in domains like speech and meeting transcript processing, where granular sentence-level labels are rarely available. To train an end-to-end speech summarizer, Takatomo Kano et al. calculated the BERTScore between source utterances and abstractive summaries to synthesize pseudo-extractive targets, guiding the model to recognize content importance. Similarly, Zhongfen Deng et al. mapped raw transcript sentences to human summaries via semantic similarity metrics, generating the pseudo-labels necessary to train a highly effective sentence-level classifier that filters out conversational noise.

The mechanism also scales to long-context retrieval-augmented generation (RAG). Haozhen Zhang et al. showed that semantic similarity metrics can effectively bridge the supervision gap when explicit ground-truth labels for document chunks are missing, allowing graph neural networks to learn complex document correlations through self-supervised ranking.

Distant Supervision in Dialogue and Knowledge Retrieval

In conversational systems, determining which historical turn or external fact directly informs the next response is a major challenge. Rather than relying on manual annotations of these dependencies, the target response itself can serve as a distant supervisor.

[Context Utterances] \
[Context Utterances]  --> Compute Semantic Similarity --> Pseudo-Labels for Relevance
[Context Utterances] /             ^
                                   |
                           [Target Response] (Oracle)

The authors of these systems use pre-trained embedding models to calculate the semantic distance between historical context and the final output:

  • Context Selection: Lei Shen et al. utilized pre-trained sentence embeddings to measure the similarity between historical utterances and the target response. This similarity score acts as a global guidance signal, supervising the model's attention mechanism to prioritize relevant context turns.
  • Background Knowledge Selection: Pengjie Ren et al. automatically generated training signals for background-grounded chatbots by calculating the similarity between the target response and segments of background text, proving that automated similarity metrics can effectively replace expensive human-labeled spans.
  • Fact Ranking: Peng Xu et al. trained a relevance ranker for story generation by using sentence embeddings to measure the similarity between retrieved external facts and the target text, eliminating the need for explicit fact-to-sentence mappings.
  • Medical Dialogue: In specialized medical dialogues, Yu Zhao 0043 et al. identified pivotal historical utterances by calculating their semantic similarity to the ground-truth medical response, establishing a robust target for an intermediate recall phase during training.

Generalizing to Multimodal and Continuous Tasks

The core mechanism of similarity-driven weak supervision extends far beyond text selection, proving highly effective for multimodal grounding, domain adaptation, and continuous regression tasks.

Multimodal Grounding

In image captioning, Chenxi Liu et al. introduced a method to supervise attention maps by calculating the semantic similarity between caption terms and known object classes. This allowed the model to learn semantically correct visual grounding without requiring expensive, manual word-to-bounding-box annotations. For controllable image captioning, Shunqi Mao et al. extracted pseudo-highlights from surrounding text context by calculating the cosine similarity within existing image-context-caption triplets. To align visual scene graphs with human perception, Sangwoong Yoon et al. used the distance between caption embeddings as a continuous supervision signal, successfully training graph neural networks without direct human graph labels.

Domain Adaptation and Seed-Based Learning

In low-resource legal decision-making, Hui Liu et al. achieved data augmentation by mapping expert-annotated segments to large unlabeled legal corpora using cosine similarity, generating weak labels at scale for stable and explainable model training. For aspect detection in product reviews, Thi-Nhung Nguyen et al. bypassed manual labels entirely by generating initial training signals from the semantic similarity between input text and a small set of predefined seed words.

Continuous Property Estimation

Finally, similarity-based pseudo-labeling can be used to capture continuous, non-binary properties. Katherine Thai et al. developed a system to quantify the precise extent of AI editing in human text. In the absence of ground-truth intensity labels, the authors utilized embedding cosine distance and n-gram overlap between original and edited versions as a proxy for the structural and stylistic changes introduced by the AI model, creating a continuous training signal for regression.

Key Foundational Work

Modeling Reportable Events as Turning Points in Narrative (2015) — contributed 1 concept elements to this direction.
Do Automatic Annotation Techniques Have Any Impact on Supervised Complex Question Answering? (2009) — contributed 1 concept elements to this direction.
Finding Short Definitions of Terms on Web Pages (2009) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_similarity_based_pseudo_labeling_e1ba16cf,
  title = {similarity-based pseudo-labeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/similarity-based-pseudo-labeling/rdr_e1ba16cf050d44a39c92de7605046280},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_similarity_based_pseudo_labeling_e1ba16cf_v1,
  title = {similarity-based pseudo-labeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/similarity-based-pseudo-labeling/rdr_e1ba16cf050d44a39c92de7605046280/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake