iUBC at SemEval-2016 Task 2: RNNs and LSTMs for interpretable STS

Iñigo Lopez-Gazpio, Eneko Agirre, Montse Maritxalar

Introduction

Standard Semantic Textual Similarity (STS) tasks typically produce a single numerical score to represent the degree of similarity between two text snippets. While useful, this "black box" approach lacks transparency. Interpretable STS (iSTS) addresses this by requiring systems to explain the relationship between sentences through chunking, alignment, and specific relationship labeling. The iUBC system, developed for SemEval-2016 Task 2, introduces a neural architecture designed to handle these multi-task requirements simultaneously.

The iUBC Pipeline

The iUBC system employs a three-stage pipeline to process sentence pairs:

  1. Chunking: Sentences are broken into meaningful segments using standard NLP tools (NLTK and ixa-pipes-chunker) supplemented by custom regular expressions.
  2. Alignment: A token-to-token similarity matrix is constructed using GloVe word embeddings and lemmatized overlap. The Hungarian-Munkres algorithm then identifies the optimal mapping between chunks.
  3. Joint Classification and Scoring: This core stage utilizes recurrent neural networks to process aligned segments.

Joint Classification and Scoring

The innovation of the iUBC system lies in its joint training approach. The architecture consists of a bottom layer—either a standard RNN or an LSTM—that generates a fixed-length vector representation of the segment. On top of this shared layer, two separate feedforward networks function as "heads": a regressor for predicting similarity scores (0-5) and a classifier for predicting relationship labels (e.g., equivalence, opposition).

The iUBC system architecture featuring a shared RNN/LSTM layer with dual regressor and classifier heads.

By backpropagating error gradients from both tasks into the shared recurrent layer, the model is forced to learn features that are robust for both classification and regression. This joint training acts as a form of regularization, improving the generalizability of the underlying representations.

Results and Insights

The system was evaluated on diverse datasets, including image descriptions, news headlines, and student answers. A key finding was that LSTMs consistently outperformed standard RNNs, demonstrating smoother learning curves and better data fitting, particularly for the more complex labeling task.

Performance comparison between RNN and LSTM models for labeling and scoring tasks over 10 epochs.

As shown in the training curves, the system generally achieved higher accuracy in predicting similarity scores than in identifying specific relationship labels. This disparity highlights the inherent difficulty of distinguishing nuanced semantic relations like "specialization" versus "similarity," especially with limited training data. Despite these challenges, iUBC achieved the top performance in the "Answer-Students" dataset for the 2016 task, proving that a neural-first approach can be highly competitive without extensive manual feature engineering.

Conclusion

The iUBC system demonstrates the effectiveness of joint neural architectures for interpretable STS. By leveraging shared representations for classification and regression, the model provides both a similarity score and an explanatory label. Future work may involve incorporating lexicalized features to improve label prediction accuracy and exploring larger datasets to further refine the recurrent layers.

Summary updated

BibTeX
@inproceedings{lopezgazpio2016iubc,
  author = {Iñigo Lopez-Gazpio and Eneko Agirre and Montse Maritxalar},
  title = {{iUBC at SemEval-2016 Task 2: RNNs and LSTMs for interpretable STS}},
  booktitle = {SemEval @ NAACL},
  year = {2016},
  doi = {10.18653/v1/s16-1119},
}
Figure