Predicate-Argument Based Bi-Encoder for Paraphrase Identification

Qiwei Peng, David J. Weir, Julie Weeds, Yekun ChaiACL 2022 · OpenReview

Introduction

Paraphrase identification is a cornerstone of natural language processing, requiring models to determine if two sentences convey the same meaning. Current approaches generally fall into two categories: high-accuracy but computationally intensive cross-encoders and highly efficient but "structure-blind" bi-encoders like Sentence-BERT (SBERT).

While bi-encoders are preferred for large-scale retrieval tasks due to their speed, their reliance on simple mean-pooling often leads to failures in distinguishing sentences with identical word counts but different semantic structures (e.g., "The dog bit the man" vs. "The man bit the dog"). Previous attempts to introduce structural awareness using Graph Convolutional Networks (RGCNs) have added millions of parameters, complicating the model without guaranteed consistency. This paper introduces a lightweight, predicate-argument based bi-encoder that adds only 3,840 parameters while significantly improving structural awareness.

Methodology: The PAS+SBERT Architecture

The proposed model, PAS+SBERT, explicitly incorporates predicate-argument information to capture the "who did what to whom" logic of a sentence.

The PAS+SBERT architecture showing the extraction of predicate-argument spans and their aggregation into a structure-aware embedding.

The architecture follows a multi-step process:

  1. PAS Extraction: An off-the-shelf Semantic Role Labeling (SRL) tagger identifies predicates and their associated arguments.
  2. Span-Based Representation: Words are grouped into predicate-argument spans. For instance, in the sentence "Rhode Island becomes the 10th state to enact marriage equality," spans are formed around the predicates "becomes" and "enact."
  3. Weighted Aggregation: A self-attention mechanism learns the relative importance of these spans, which are then aggregated into a "Span-Based Sentence Embedding."
  4. Hybrid Embedding: The final sentence representation is a concatenation of the standard SBERT mean-pooled vector and the new PAS-based vector.

This approach is remarkably efficient, adding a mere 3,840 parameters—a tiny fraction compared to the ~32 million parameters typically added by RGCN-based structural models.

Experimental Setup

The model was evaluated across six diverse benchmarks to test its robustness:

  • MSRP: News article clusters.
  • TwitterURL & PIT2015: Informal social media data.
  • QQP (Quora Question Pairs): Large-scale community question data.
  • PAWS (Wiki & QQP): Adversarial datasets featuring high word overlap but different meanings, specifically designed to challenge structure-blind models.

Results and Discussion

The PAS+SBERT model outperformed standard SBERT and SRoBERTa on five of the six datasets. Notably, it showed the most significant gains on the adversarial PAWS datasets and the PIT2015 dataset (where F1 scores jumped from 52.03 to 59.19).

Performance comparison between SBERT and PAS+SBERT across various training data percentages on six datasets.

A key finding is the model's performance in low-data regimes. As shown in the figure above, the PAS-enhanced model maintains a substantial lead over standard SBERT even when trained on only 10% or 30% of the available data. This suggests that the predicate-argument structure provides a strong inductive bias that helps the model generalize better with fewer examples. Furthermore, PAS+SBERT outperformed the much heavier SBERT-RGCN model on nearly every metric, proving that structural awareness does not require massive parameter overhead.

Conclusion and Future Work

By focusing on the core logical structure of sentences through predicate-argument spans, the researchers have developed a highly efficient way to bolster bi-encoders against structural confusion. While the model currently depends on the quality of external SRL taggers and does not yet utilize specific semantic role labels (like "Agent" or "Patient"), it represents a significant step toward lightweight, structure-aware NLP. Future work may involve incorporating these explicit role tags and extending the architecture to other sentence-pair tasks such as Natural Language Inference (NLI).

Summary updated

BibTeX
@inproceedings{peng2022predicate,
  author = {Qiwei Peng and David J. Weir and Julie Weeds and Yekun Chai},
  title = {{Predicate-Argument Based Bi-Encoder for Paraphrase Identification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=87wz2BNloqB},
}
Figure