Introduction
Sentence embeddings are fundamental components in modern Natural Language Processing (NLP), enabling machines to understand semantic relationships for tasks like document clustering, information retrieval, and sentiment analysis. While supervised models like Sentence-BERT (SBERT) achieve high performance, they rely on vast amounts of manually labeled data—a resource often unavailable in specialized or industrial domains. Conversely, unsupervised methods, while data-rich, frequently lack the precision required for complex semantic tasks.
PAUSE (Positive and Annealed Unlabeled Sentence Embedding) introduces a framework designed to bridge this gap. By leveraging Positive-Unlabeled (PU) learning and a novel annealing strategy, PAUSE learns high-quality embeddings from partially labeled datasets, maintaining competitive performance even when labels are scarce.
The Challenge of Sparse Labels
In many practical scenarios, researchers encounter a "Positive-Unlabeled" (PU) data distribution. We might have a small set of confirmed positive examples (e.g., sentence pairs that definitely entail one another) but a massive corpus of unlabeled pairs. Standard supervised learning often fails here because it treats unlabeled data as negative, introducing significant noise and bias.
PAUSE addresses this by treating unlabeled samples as a weighted mixture of potential classes, allowing the model to extract signal from the entire dataset without the prohibitive cost of full manual annotation.
Methodology: Architecture and Annealing
The PAUSE framework utilizes a dual-encoder architecture, typically initialized with a pre-trained model like BERT.
Dual-Encoder and Multi-Class PU Loss
As shown in the architecture diagram, PAUSE processes a premise ($x_i$) and a hypothesis ($x'_i$) through shared encoders. The resulting sentence embeddings, $g(x_i)$ and $g(x'_i)$, are combined using concatenation, absolute difference, and element-wise product. This feature vector is then passed through fully-connected layers to a $C$-way classifier.
To handle unlabeled data, the authors extended binary PU learning to a multi-class setting. Each class is treated as a binary sub-problem where the "PU loss" considers unlabeled samples as potentially belonging to any class, weighted by their estimated prior probability.
Polynomial Annealing
A critical innovation in PAUSE is the use of Polynomial Annealing. Early in the training process, the model's risk estimates for unlabeled data are unstable. To prevent the model from converging on incorrect signals, the influence of the PU loss is gradually introduced using an annealing factor $(t/T)^\alpha$. This allows the model to first stabilize on the high-quality labeled data before incorporating the noisier unlabeled signals, ensuring robust convergence.
Experimental Results
The researchers evaluated PAUSE across standard benchmarks, including Natural Language Inference (NLI) and Semantic Textual Similarity (STS) datasets.
The 10% Efficiency Threshold
One of the most striking findings is the model's label efficiency. As detailed in the results table, PAUSE-NLI-base-10% (using only 10% of available labels) achieved an average Spearman’s correlation of 74.68 across STS tasks. This performance is remarkably close to the fully supervised SBERT-NLI-base (74.89), which uses 100% of the labels.
Real-World Application
In an industrial "Similar Company" dataset, PAUSE demonstrated similar efficiency. It achieved 73.35% accuracy with only 10% of labels, compared to 76.13% when using the full labeled set. This suggests that for many applications, the marginal utility of labeling the remaining 90% of data is low when using the PAUSE framework.
Conclusion
PAUSE demonstrates that high-quality sentence embeddings do not always require massive labeled datasets. By effectively integrating unlabeled data through multi-class PU learning and stabilizing the process with polynomial annealing, it offers a cost-effective solution for domain-specific NLP tasks. While performance degrades when labels fall below 5%, the ability to reach near-state-of-the-art results with only 10% of the data marks a significant step forward in label-efficient deep learning.
Summary updated