SetCSE: Set Operations using Contrastive Learning of Sentence Embeddings

Kang Liu

Abstract

Taking inspiration from Set Theory, we introduce SetCSE, an innovative information retrieval framework. SetCSE employs sets to represent complex semantics and incorporates well-defined operations for structured information querying under the provided context. Within this framework, we introduce an inter-set contrastive learning objective to enhance comprehension of sentence embedding models concerning the given semantics. Furthermore, we present a suite of operations, including SetCSE intersection, difference, and operation series, that leverage sentence embeddings of the enhanced model for complex sentence retrieval tasks. Throughout this paper, we demonstrate that SetCSE adheres to the conventions of human language expressions regarding compounded semantics, provides a significant enhancement in the discriminatory capability of underlying sentence embedding models, and enables numerous information retrieval tasks involving convoluted and intricate prompts which cannot be achieved using existing querying methods.

Introduction

Modern information retrieval (IR) and semantic search systems typically rely on single-sentence queries. While effective for simple lookups, this paradigm struggles with "convoluted" or "intricate" semantics that are difficult to encapsulate in a single prompt. For example, a multifaceted concept like "Environmental, Social, and Governance (ESG)" is more naturally defined through a collection of representative examples rather than a solitary definition.

Furthermore, standard IR lacks a formal syntax for logical filtering based on these complex semantics. Users often need to perform operations like "finding sentences about technology in social contexts while excluding privacy breaches." SetCSE addresses these gaps by treating complex semantics as sets of sentences and providing a mathematically grounded framework for logical operations like intersection and difference.

Methodology

The SetCSE framework operates in two distinct phases: training and retrieval.

1. Inter-set Contrastive Learning

To improve the model's ability to distinguish between related but distinct concepts, we propose a fine-tuning objective that treats different sets of sentences as negative pairs. By minimizing this "inter-set loss," the underlying encoder (e.g., RoBERTa or BERT) learns to push different semantic clusters apart in the vector space while pulling similar examples within a set closer together.

2. SetCSE Operations

Once the model is enhanced, we define retrieval operations based on the average cosine similarity between a candidate sentence and the target sets:

  • Intersection ($A \cap B$): Ranks sentences in set $A$ based on their similarity to the semantic profile of set $B$.
  • Difference ($A \setminus C$): Ranks sentences in $A$ based on their dissimilarity to set $C$, effectively allowing for "semantic subtraction."
  • Serial Operations: These can be chained (e.g., $U \cap B \cap D \setminus E$) to execute highly specific queries, such as isolating technical social issues while filtering out danger-related content.
Overview of SetCSE inter-set contrastive learning (left) and the execution of a serial set operation for retrieval (right).

Evaluation and Results

We evaluated SetCSE across diverse domains, including topic classification (AG News), sentiment analysis (Financial PhraseBank), and intent detection (Banking77).

Performance and Discriminatory Capability

SetCSE demonstrated massive performance gains, improving accuracy in "Intersection" tasks by an average of 39% and "Difference" tasks by 14-21% compared to state-of-the-art embedding models. Qualitative analysis via t-SNE visualizations confirms that SetCSE significantly enhances the clustering of semantic categories, making them much more linearly separable than existing methods.

t-SNE visualizations comparing existing embeddings with SetCSE-enhanced embeddings. SetCSE creates much tighter, more distinct clusters for Business, SciTech, and Sports topics.

Robustness and Model Architecture

Crucially, this specialized fine-tuning does not destroy the model's general capabilities; we observed only a minor (~7%) decrease in performance on standard Semantic Textual Similarity (STS) benchmarks. We also found that encoder-only models (like RoBERTa) significantly outperform decoder-only models (like SGPT) in this framework, likely due to the importance of bidirectional context in capturing set-based semantics.

Conclusion

SetCSE provides a robust, mathematically grounded approach to complex information retrieval. By leveraging the collective power of sentence sets and inter-set contrastive learning, it enables users to perform logical semantic queries that were previously impossible. Future work will focus on adapting this framework for Large Language Models (LLMs) and exploring parameter-efficient fine-tuning (PEFT) methods like LoRA to reduce the computational overhead of the training phase.

Summary updated

Abstract

Taking inspiration from Set Theory, we introduce SetCSE, an innovative information retrieval framework. SetCSE employs sets to represent complex semantics and incorporates well-defined operations for structured information querying under the provided context. Within this framework, we introduce an inter-set contrastive learning objective to enhance comprehension of sentence embedding models concerning the given semantics. Furthermore, we present a suite of operations, including SetCSE intersection, difference, and operation series, that leverage sentence embeddings of the enhanced model for complex sentence retrieval tasks. Throughout this paper, we demonstrate that SetCSE adheres to the conventions of human language expressions regarding compounded semantics, provides a significant enhancement in the discriminatory capability of underlying sentence embedding models, and enables numerous information retrieval tasks involving convoluted and intricate prompts which cannot be achieved using existing querying methods.
BibTeX
@inproceedings{liu2024setcse,
  author = {Kang Liu},
  title = {{SetCSE: Set Operations using Contrastive Learning of Sentence Embeddings}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=zEHGSN8Hy8},
}
Figure