PAUSE: Positive and Annealed Unlabeled Sentence Embedding

Lele Cao, Emil Larsson, Vilhelm von Ehrenheim, Dhiana Deva Cavalcanti Rocha, Anna Martin, Sonja Horn

Introduction

Sentence embeddings are fundamental components in modern Natural Language Processing (NLP), enabling machines to understand semantic relationships for tasks like document clustering, information retrieval, and sentiment analysis. While supervised models like Sentence-BERT (SBERT) achieve high performance, they rely on vast amounts of manually labeled data—a resource often unavailable in specialized or industrial domains. Conversely, unsupervised methods, while data-rich, frequently lack the precision required for complex semantic tasks.

PAUSE (Positive and Annealed Unlabeled Sentence Embedding) introduces a framework designed to bridge this gap. By leveraging Positive-Unlabeled (PU) learning and a novel annealing strategy, PAUSE learns high-quality embeddings from partially labeled datasets, maintaining competitive performance even when labels are scarce.

The Challenge of Sparse Labels

In many practical scenarios, researchers encounter a "Positive-Unlabeled" (PU) data distribution. We might have a small set of confirmed positive examples (e.g., sentence pairs that definitely entail one another) but a massive corpus of unlabeled pairs. Standard supervised learning often fails here because it treats unlabeled data as negative, introducing significant noise and bias.

PAUSE addresses this by treating unlabeled samples as a weighted mixture of potential classes, allowing the model to extract signal from the entire dataset without the prohibitive cost of full manual annotation.

Methodology: Architecture and Annealing

The PAUSE framework utilizes a dual-encoder architecture, typically initialized with a pre-trained model like BERT.

The PAUSE dual-encoder architecture processes sentence pairs through shared weights and multiple matching strategies.

Dual-Encoder and Multi-Class PU Loss

As shown in the architecture diagram, PAUSE processes a premise ($x_i$) and a hypothesis ($x'_i$) through shared encoders. The resulting sentence embeddings, $g(x_i)$ and $g(x'_i)$, are combined using concatenation, absolute difference, and element-wise product. This feature vector is then passed through fully-connected layers to a $C$-way classifier.

To handle unlabeled data, the authors extended binary PU learning to a multi-class setting. Each class is treated as a binary sub-problem where the "PU loss" considers unlabeled samples as potentially belonging to any class, weighted by their estimated prior probability.

Polynomial Annealing

A critical innovation in PAUSE is the use of Polynomial Annealing. Early in the training process, the model's risk estimates for unlabeled data are unstable. To prevent the model from converging on incorrect signals, the influence of the PU loss is gradually introduced using an annealing factor $(t/T)^\alpha$. This allows the model to first stabilize on the high-quality labeled data before incorporating the noisier unlabeled signals, ensuring robust convergence.

Experimental Results

The researchers evaluated PAUSE across standard benchmarks, including Natural Language Inference (NLI) and Semantic Textual Similarity (STS) datasets.

Performance comparison on STS tasks shows PAUSE-base-10% competing with fully supervised models.

The 10% Efficiency Threshold

One of the most striking findings is the model's label efficiency. As detailed in the results table, PAUSE-NLI-base-10% (using only 10% of available labels) achieved an average Spearman’s correlation of 74.68 across STS tasks. This performance is remarkably close to the fully supervised SBERT-NLI-base (74.89), which uses 100% of the labels.

Real-World Application

In an industrial "Similar Company" dataset, PAUSE demonstrated similar efficiency. It achieved 73.35% accuracy with only 10% of labels, compared to 76.13% when using the full labeled set. This suggests that for many applications, the marginal utility of labeling the remaining 90% of data is low when using the PAUSE framework.

Conclusion

PAUSE demonstrates that high-quality sentence embeddings do not always require massive labeled datasets. By effectively integrating unlabeled data through multi-class PU learning and stabilizing the process with polynomial annealing, it offers a cost-effective solution for domain-specific NLP tasks. While performance degrades when labels fall below 5%, the ability to reach near-state-of-the-art results with only 10% of the data marks a significant step forward in label-efficient deep learning.

Summary updated

Abstract

Sentence embedding refers to a set of effective and versatile techniques for converting raw text into numerical vector representations that can be used in a wide range of natural language processing (NLP) applications. The majority of these techniques are either supervised or unsupervised. Compared to the unsupervised methods, the supervised ones make less assumptions about optimization objectives and usually achieve better results. However, the training requires a large amount of labeled sentence pairs, which is not available in many industrial scenarios. To that end, we propose a generic and end-to-end approach -- PAUSE (Positive and Annealed Unlabeled Sentence Embedding), capable of learning high-quality sentence embeddings from a partially labeled dataset. We experimentally show that PAUSE achieves, and sometimes surpasses, state-of-the-art results using only a small fraction of labeled sentence pairs on various benchmark tasks. When applied to a real industrial use case where labeled samples are scarce, PAUSE encourages us to extend our dataset without the liability of extensive manual annotation work.
BibTeX
@inproceedings{cao2021pause,
  author = {Lele Cao and Emil Larsson and Vilhelm von Ehrenheim and Dhiana Deva Cavalcanti Rocha and Anna Martin and Sonja Horn},
  title = {{PAUSE: Positive and Annealed Unlabeled Sentence Embedding}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.791},
}
Figure