SetCSE: Set Operations using Contrastive Learning of Sentence Embeddings

Kang Liu

Introduction

Modern sentence embedding models, such as BERT, RoBERTa, and SimCSE, excel at standard "semantic search"—finding a sentence in a corpus that is most similar to a single query string. However, human language and intent are often far more complex. We frequently define intricate concepts not with a single sentence, but through a collection of examples. For instance, the concept of "Environmental, Social, and Governance" (ESG) is difficult to capture in one prompt; it is much better defined by a group of related topics.

Existing information retrieval (IR) methods struggle with these "compounded semantics." They lack a formal syntax to handle structured, multi-faceted queries like: "Find me sentences about technology used for social good, but exclude those that mention negative environmental impacts."

To address this, researchers have introduced SetCSE, a novel framework that brings the logic of Set Theory to the world of natural language processing. By representing complex concepts as sets of sentences and using a specialized contrastive learning objective, SetCSE allows researchers to perform structured semantic queries with significantly higher precision than standard embedding models.

Methods: Inter-set Learning and Set-Theoretic Operations

SetCSE tackles the problem of compounded semantics through two major algorithmic innovations:

1. Inter-set Contrastive Learning

Standard contrastive learning typically pulls "positive" pairs (similar sentences) together and pushes "negative" pairs apart. SetCSE introduces a novel loss function ($\mathcal{L}_{inter-set}$) that treats entire sets of sentences as distinct semantic units. During fine-tuning, the model learns to differentiate between these sets, effectively "sharpening" the boundaries between different concepts in the embedding space. This acts like a focus adjustment for a camera: where standard models see semantic clusters as overlapping clouds, SetCSE forces the model to recognize the gaps between them.

2. Set-Theoretic Operations

The framework defines "semantic closeness" as the average similarity between a candidate sentence and all members of a target set. This mathematical foundation allows for the definition of formal set operations:

  • Intersection ($A \cap B$): Ranking sentences in set $A$ based on their similarity to set $B$.
  • Difference ($A \setminus C$): Ranking sentences in $A$ based on their dissimilarity to set $C$ (deselection).
  • Operation Series: Combining these into complex chains (e.g., $A \cap B_1 \cap B_2 \setminus C_1$) to perform highly specific filtering.
Illustration of a SetCSE operation series performing complex semantic filtering.

Crucially, the algorithm is a two-step process: first, fine-tune the embedding model on the provided context sets using the inter-set objective; second, rank the target data using the defined set operations.

Key Insights and Applications

The core "aha moment" of SetCSE is that a set of sentences is a fundamentally more robust way to define a "fuzzy" concept than a single vector. By using multiple examples, the model captures the breadth of a topic rather than just a single point in vector space.

This has immediate practical applications, particularly in Active Learning. The paper demonstrates that an operation like $X \cap S_1 \cap S_2$ can identify sentences sitting exactly on the decision boundary between two classes. This is a highly valuable tool for data labeling, as these ambiguous sentences are the most informative for humans to annotate.

Key Results

The authors evaluated SetCSE across a diverse array of benchmarks—including AG News (Topic Classification), Financial PhraseBank (Finance), and Banking77 (Intent Detection)—to ensure domain-agnostic performance.

  • Massive Accuracy Gains: On average, SetCSE improved semantic awareness by 30-38% over existing models. For the "Intersection" task, improvements were remarkably high, reaching up to 56% on specific datasets like AG News.
Table detailing the massive accuracy and F1 score improvements of SetCSE over existing models across multiple datasets.
  • Visual Separation: t-SNE visualizations of high-dimensional data show that after SetCSE fine-tuning, semantic clusters that were previously tangled together become clearly separated and distinct.
  • Complex Query Success: In real-world tests on S&P 500 earnings calls, SetCSE successfully extracted nuanced sentences regarding "using technology to solve social issues while neglecting negative impacts"—a task that would likely fail with simple keyword or single-prompt semantic searches.

Limitations and Future Work

While SetCSE represents a significant leap forward for structured semantic retrieval, the authors note a few limitations:

  • Computational Scaling: The number of negative pairs in the loss function grows quadratically with the number of sets ($N$). While random sampling mitigates this, it could still be a bottleneck for extremely large-scale semantic taxonomies.
  • Non-Commutativity: Unlike pure math, SetCSE operations are not commutative ($A \cap B \neq B \cap A$). The first set is always the "search space" and subsequent sets are "filters." While intuitive for IR, it diverges from classical set theory.
  • Model Scope: The study focused on encoder-only models (like BERT) and smaller decoder models (SGPT-125M). Performance on massive Large Language Models (LLMs) remains an open area for research.

Future directions include integrating Low-Rank Adaptation (LoRA) to make the fine-tuning step even more efficient and developing a standardized API for "Set-SQL" style querying.

Summary updated

Abstract

Taking inspiration from Set Theory, we introduce SetCSE, an innovative information retrieval framework. SetCSE employs sets to represent complex semantics and incorporates well-defined operations for structured information querying under the provided context. Within this framework, we introduce an inter-set contrastive learning objective to enhance comprehension of sentence embedding models concerning the given semantics. Furthermore, we present a suite of operations, including SetCSE intersection, difference, and operation series, that leverage sentence embeddings of the enhanced model for complex sentence retrieval tasks. Throughout this paper, we demonstrate that SetCSE adheres to the conventions of human language expressions regarding compounded semantics, provides a significant enhancement in the discriminatory capability of underlying sentence embedding models, and enables numerous information retrieval tasks involving convoluted and intricate prompts which cannot be achieved using existing querying methods.
BibTeX
@inproceedings{liu2024setcse,
  author = {Kang Liu},
  title = {{SetCSE: Set Operations using Contrastive Learning of Sentence Embeddings}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=zEHGSN8Hy8},
}
Figure