Varying Sentence Representations via Condition-Specified Routers

Ziyong Lin, Quansen Wang, Zixia Jia, Zilong ZhengEMNLP 2024 · OpenReview

Abstract

Semantic similarity between two sentences is inherently subjective and can vary significantly based on the specific aspects emphasized. Consequently, traditional sentence encoders must be capable of generating conditioned sentence representations that account for diverse conditions or aspects. In this paper, we propose a novel yet efficient framework based on transformer-style language models that facilitates advanced conditioned sentence representation while maintaining model parameters and computational efficiency. Empirical evaluations on the Conditional Semantic Textual Similarity and Knowledge Graph Completion tasks demonstrate the superiority of our proposed framework.

Introduction

Standard Sentence Textual Similarity (STS) models typically generate a single, static embedding for a sentence, regardless of the context in which it is being compared. However, similarity is inherently subjective. Consider two descriptions: "A hotel room decorated in silver and white" and "A small bedroom suite in a hotel with a bed, table, and two chairs." If the condition for comparison is "the name of the place," these sentences are highly similar; however, if the condition is "the number of chairs," they are not.

Existing approaches to this problem generally fall into two categories: Tri-Encoders and Cross-Encoders. Tri-Encoders encode sentences and conditions separately, offering high speed but lacking the interaction necessary to capture nuanced, condition-specific semantics. Conversely, Cross-Encoders process all inputs simultaneously, achieving high accuracy at the cost of extreme computational overhead, making them impractical for large-scale retrieval. This work introduces the Condition-Specified Router (CSR) to bridge this gap, providing Cross-Encoder-level nuance with Tri-Encoder efficiency.

Methodology

The proposed framework, Conditioned Sentence Representation (CSR), integrates a routing mechanism into the final layers of a standard Transformer-style encoder (e.g., BERT or SimCSE). Unlike previous methods that merge embeddings only at the output stage, CSR allows the condition to influence the encoding process itself.

Comparison of Cross-Encoder, Bi-Encoder, Tri-Encoder, and the proposed CSR architecture.

The Condition-Specified Router Mechanism

The CSR mechanism functions as a "spotlight" that highlights relevant tokens within a sentence based on a given condition. The process involves four key steps:

  1. Query Generation: The [CLS] token from the condition sentence is extracted to serve as a query vector ($q^c$).
  2. Key Extraction: The model identifies key vectors ($K^s$) for every token in the target sentence.
  3. Routing: A "router score" is calculated for each token by measuring the compatibility between the condition query and the sentence keys.
  4. Scaling: The hidden states of relevant tokens are enhanced by a factor of $(1 + \text{router score})$.
Detailed architecture of the Condition-Specified Router (CSR) within the Transformer blocks.

A critical advantage of this approach is that it is parameter-free. It utilizes the existing weights of the pre-trained model, requiring no additional learnable parameters. This makes CSR highly efficient and easily adaptable to existing architectures.

Key Insights

  • Optimal Placement: Empirical results suggest that the router is most effective when placed in the late stages of the model (specifically the penultimate layer). This allows the model to first establish a general semantic context before applying the condition-specific filter.
  • Efficiency of Parameter-Free Design: Interestingly, adding dedicated learnable weights to the router (a "Heavy Router") did not improve performance. This indicates that pre-trained Transformers already possess the internal representations necessary for effective routing; they simply require a mechanism to trigger them.
  • Context Preservation: The scaling formula $(1 + w_t)$ ensures that while relevant tokens are emphasized, non-relevant tokens are not entirely discarded, thus preserving the overall grammatical and semantic integrity of the sentence representation.

Experimental Results

The CSR framework was evaluated across two primary benchmarks:

  • C-STS (Conditional Semantic Textual Similarity): CSR demonstrated a significant performance boost, achieving a 6.7% to 7.1% improvement over baseline Tri-Encoder models.
  • Knowledge Graph Completion (KGC): In this task, CSR nearly matched the accuracy of the much slower Bi-Encoder architecture. Notably, it outperformed competing methods like Hyper-CL while using significantly fewer parameters and maintaining higher throughput.
  • Zero-Shot Performance: Even without task-specific fine-tuning, the CSR mechanism outperformed much larger models, such as Llama-2-7B, in zero-shot conditional similarity tasks, highlighting its inherent ability to align sentence tokens with condition semantics.

Conclusion and Future Work

The Condition-Specified Router provides a scalable and efficient solution for generating aspect-aware sentence embeddings. While highly effective for encoder-based models in STS and KGC tasks, future research will explore its application to other conditioned tasks like Aspect-Based Sentiment Analysis (ABSA). Additionally, adapting the CSR mechanism for decoder-only architectures (e.g., GPT-style models) remains an ongoing challenge due to their unidirectional attention constraints.

Summary updated

Abstract

Semantic similarity between two sentences is inherently subjective and can vary significantly based on the specific aspects emphasized. Consequently, traditional sentence encoders must be capable of generating conditioned sentence representations that account for diverse conditions or aspects. In this paper, we propose a novel yet efficient framework based on transformer-style language models that facilitates advanced conditioned sentence representation while maintaining model parameters and computational efficiency. Empirical evaluations on the Conditional Semantic Textual Similarity and Knowledge Graph Completion tasks demonstrate the superiority of our proposed framework.
BibTeX
@inproceedings{lin2024varying,
  author = {Ziyong Lin and Quansen Wang and Zixia Jia and Zilong Zheng},
  title = {{Varying Sentence Representations via Condition-Specified Routers}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=RGVpXA1Z4m},
}
Figure