VAEGPT-Sim: Improving Sentence Representation with Limited Corpus Using Gradually-Denoising VAE

Zhenyi Wang, Haiyan Ning, Qing Ling, Dan WangACL 2024 · DOI

Abstract

High-quality text embeddings are essential for modern NLP tasks but typically require massive datasets for training. This paper introduces VAEGPT-Sim, a generative model that combines a denoising Variational Autoencoder (VAE) with a GPT-2 backbone to produce diverse, high-quality synonyms for data augmentation. By employing a gradually-denoising strategy and a self-judgment loss, VAEGPT-Sim enables effective contrastive learning in data-constrained environments. Experimental results show that models trained on just 70k sentences using VAEGPT-Sim can outperform state-of-the-art methods trained on millions of sentences, particularly in specialized domains like biomedicine.

Introduction

The efficacy of semantic search, document clustering, and other downstream NLP tasks relies heavily on the quality of text embeddings. Current state-of-the-art contrastive learning frameworks, such as SimCSE and DiffCSE, are notoriously data-hungry, often requiring over a million sentences to achieve peak performance. This requirement poses a significant challenge for specialized domains—such as legal or biomedical fields—where large-scale labeled corpora are scarce and general-purpose models fail to capture domain-specific nuances.

Traditional data augmentation techniques, like random word deletion or shuffling, often result in "unnatural" sentences that lack semantic depth. VAEGPT-Sim addresses this by generating high-quality synonyms that maintain the original meaning while introducing lexical variety, thereby providing superior positive pairs for contrastive learning.

Methodology

VAEGPT-Sim is a generative model designed to create high-quality synonyms for data augmentation. It features two main innovations: the integration of a gradually-denoising VAE and a self-judgment loss mechanism.

Gradually-Denoising VAE

The model integrates VAE structures into each of the 12 decoder blocks of a GPT-2 backbone. A "gradual denoising" strategy is employed where Gaussian noise is added to the hidden states. The intensity of this noise decreases as the signal progresses through deeper layers, controlled by a temperature coefficient. This mechanism allows the model to explore a diverse latent space in early layers while ensuring the final output is coherent and semantically grounded.

Architecture of the VAEGPT-Sim model showing the training loss and the internal block structure.

Self-Judgment Loss

To maintain semantic consistency, the authors implement a self-judgment mechanism. A weight-frozen GPT-2 acts as a discriminator, calculating the cosine similarity between the embeddings of the generated sentence and the target sentence. This "target loss" guides the generator to produce synonyms that are semantically equivalent to the source text.

These generated synonyms are integrated into the Generate-CSE framework. During training, the system randomly chooses between simple augmentations (e.g., shuffling, cutoffs) and VAEGPT-Sim generations to create positive pairs for training a BERT encoder.

Experimental Setup

The research utilized several datasets to evaluate the model's performance:

  • Generator Fine-tuning: SICK (9.4k pairs) and MRPC (5.8k pairs).
  • Representation Training: 70.4k sentences from STS and NLI datasets.
  • Evaluation Benchmarks: Standard Semantic Textual Similarity (STS) tasks from 2012–2016 and the STS Benchmark (STS-B).
  • Domain-Specific Testing: 55k professional sentences from PubMed were used to evaluate performance in the biomedical domain.

Results and Discussion

VAEGPT-Sim demonstrates significant improvements in sentence representation, especially when training data is limited.

STS Benchmark Performance

On standard STS tasks, a BERT-base model trained with VAEGPT-Sim achieved an average Spearman’s correlation of 77.29. Notably, when restricted to a 70k sentence training set, VAEGPT-Sim reached a score of 76.42, significantly outperforming the standard SimCSE method (73.14) on the same data.

Performance comparison of VAEGPT-Sim against baseline models on various STS benchmarks.

Domain-Specific Advantage

In the biomedical domain, the model's efficiency is even more pronounced. A VAEGPT-Sim model trained on only 55k PubMed sentences achieved a correlation of 56.08, surpassing a SimCSE model trained on 1 million Wikipedia sentences (48.37). This highlights that for specialized fields, data relevance and quality are more critical than sheer volume.

Efficiency and Diversity

The model successfully navigates the trade-off between lexical diversity and semantic similarity. While many generative models produce either repetitive or irrelevant outputs, VAEGPT-Sim's latent space exploration combined with the discriminator's guardrails ensures high-quality positive pairs. Furthermore, achieving results comparable to SimCSE while using roughly 14 times less data makes it a highly practical choice for resource-constrained environments.

Conclusion

VAEGPT-Sim offers a practical solution for generating high-quality text embeddings in data-scarce environments. By achieving results comparable to state-of-the-art models with significantly less data, it lowers the barrier for developing specialized NLP applications. Future work will explore more sophisticated loss functions and investigate the model's scalability on larger datasets.

Summary updated

BibTeX
@inproceedings{wang2024vaegpt,
  author = {Zhenyi Wang and Haiyan Ning and Qing Ling and Dan Wang},
  title = {{VAEGPT-Sim: Improving Sentence Representation with Limited Corpus Using Gradually-Denoising VAE}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.513},
}
Figure