Structure-aware Sentence Encoder in Bert-Based Siamese Network

Qiwei Peng, David Weir, Julie WeedsACL 2021

TL;DR

This paper introduces a novel sentence encoder that enhances BERT's ability to understand sentence similarity by incorporating syntactic structure. By combining BERT with Graph Convolutional Networks (GCNs) in a Siamese architecture, their model achieves superior performance on semantic textual similarity tasks and transfer learning benchmarks compared to existing methods like Sentence-BERT.

What Problem Does This Paper Address?

Pre-trained language models like BERT have revolutionized Natural Language Understanding (NLU) tasks. However, when it comes to comparing sentences for similarity (Semantic Textual Similarity or STS), BERT-like models, even when fine-tuned into Sentence-BERT (SBERT), exhibit limitations. Specifically, SBERT, while a significant improvement, doesn't explicitly leverage the inherent syntactic structure of sentences. This can lead to inefficiencies and suboptimal performance, especially in unsupervised similarity comparison tasks. The paper aims to bridge this gap by developing a sentence encoder that explicitly integrates syntactic information into the embedding process, leading to more robust and accurate sentence representations.

Methods

The core of the proposed method is a Structure-aware Sentence Encoder built upon a Siamese Network architecture, inspired by SBERT. Here's a breakdown:

  • BERT as a Base: Each input sentence is first processed by a pre-trained BERT-base model. The output of BERT provides two key pieces of information:
  • A sentence representation obtained through mean-pooling over token embeddings.
  • Contextualized token representations, which are then used to initialize the Graph Convolutional Network (GCN).
  • Syntactic Structure Integration with RGCNs: To incorporate syntactic information, the authors employ Relational Graph Convolutional Networks (RGCNs).
  • Sentences are parsed into dependency trees using the spaCy parser. These trees are then represented as directed graphs.
  • The RGCN acts as a "structure encoder," processing these graphs. At each RGCN layer, node representations (corresponding to words) are updated by aggregating information from their neighbors (other words connected by dependency relations).
  • The RGCN's output is then mean-pooled to derive a structure-aware sentence representation.
  • Siamese Network: The BERT and RGCN outputs for two input sentences are concatenated and fed into a layer normalization layer to produce the final sentence embeddings. These embeddings are then interacted (using concatenation of embeddings and their absolute difference) and passed through a classifier for training. The Siamese structure ensures that parameters are shared and updated symmetrically for both sentences.
  • Training Objective: The model is trained using cross-entropy loss on Natural Language Inference (NLI) datasets (SNLI and MNLI), similar to SBERT, to produce general-purpose sentence embeddings.

Datasets

  • Training:
    • SNLI (Stanford Natural Language Inference)
    • MNLI (Multi-Genre Natural Language Inference)
  • Evaluation:
    • Unsupervised STS Tasks: STS12-16, SICK-Relatedness (SICK-R), and STS benchmark (STSb). These datasets are used to evaluate semantic textual similarity.
  • Transfer Tasks (SentEval): A suite of 8 classification tasks including sentiment analysis (MR, CR, SST-5, SST-2), question-type classification (TREC), subjectivity-objectivity (SUBJ), phrase-level opinion polarity (MPQA), and paraphrase detection (MRPC).

Key Insights and Ideas

  • Explicit Syntax Matters for Similarity: The paper strongly suggests that explicitly modeling syntactic structure, beyond what BERT implicitly captures, significantly improves sentence similarity comparison. This is a key differentiator from standard SBERT.
  • RGCNs as Effective Structure Encoders: RGCNs prove to be a powerful tool for encoding structural information from dependency trees into sentence representations.
  • Siamese Architecture for General Embeddings: The use of a Siamese network, trained on NLI data, is crucial for generating general-purpose sentence embeddings that can be applied to various downstream tasks without task-specific fine-tuning.
  • Complexity vs. Simplicity in Probing: The authors observe that simple logistic regression classifiers struggle to extract useful information from their more complex, structure-aware embeddings. However, a more powerful MLP classifier reveals the benefits of their approach, particularly on fine-grained classification tasks.

Key Results

  • Superior STS Performance: The proposed model outperforms SBERT and other baseline sentence encoders on all 7 unsupervised STS datasets, achieving the highest average Spearman rank correlation. For example, on STSb, their model achieved 78.50, compared to SBERT's 76.98.
  • Competitive Transfer Task Performance: On transfer tasks evaluated with logistic regression, the model shows comparable or slightly better performance than SBERT across most tasks.
  • Enhanced Fine-Grained Classification: When using a more powerful MLP classifier for transfer tasks, the structure-aware model demonstrates clear improvements on SST-5 (sentiment classification) and TREC (question-type classification), highlighting the benefit of syntactic supervision for these tasks.

Limitations and Future Work

  • USE on SICK-R: The paper acknowledges that their model performs worse than Universal Sentence Encoder (USE) on the SICK-R dataset. They attribute this to USE being trained on a broader range of datasets, including question-answering and online forums, which might be particularly suited for SICK-R.
  • Information Loss on Subjectivity: For the SUBJ task (subjectivity-objectivity), both SBERT and their model perform worse than BERT-AVG and BERT-CLS. This suggests that fine-tuning on NLI datasets might lead to some loss of information relevant to recognizing sentence subjectivity.
  • Probing Classifier Limitations: The authors hypothesize that the complexity of their model's embeddings might require more sophisticated probing classifiers to fully reveal their capabilities, as demonstrated by the MLP results.
  • Future Work: The paper implicitly suggests exploring the integration of semantic information alongside syntactic structure, as they experimented with semantic graphs but found them less effective than syntactic ones on their development set. Further investigation into how to best combine these different types of structural information could be a promising direction.

Summary updated

BibTeX
@inproceedings{peng2021structure,
  author = {Qiwei Peng and David Weir and Julie Weeds},
  title = {{Structure-aware Sentence Encoder in Bert-Based Siamese Network}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure