Introduction
Standard Semantic Textual Similarity (STS) tasks typically produce a single numerical score to represent the degree of similarity between two text snippets. While useful, this "black box" approach lacks transparency. Interpretable STS (iSTS) addresses this by requiring systems to explain the relationship between sentences through chunking, alignment, and specific relationship labeling. The iUBC system, developed for SemEval-2016 Task 2, introduces a neural architecture designed to handle these multi-task requirements simultaneously.
The iUBC Pipeline
The iUBC system employs a three-stage pipeline to process sentence pairs:
- Chunking: Sentences are broken into meaningful segments using standard NLP tools (NLTK and ixa-pipes-chunker) supplemented by custom regular expressions.
- Alignment: A token-to-token similarity matrix is constructed using GloVe word embeddings and lemmatized overlap. The Hungarian-Munkres algorithm then identifies the optimal mapping between chunks.
- Joint Classification and Scoring: This core stage utilizes recurrent neural networks to process aligned segments.
Joint Classification and Scoring
The innovation of the iUBC system lies in its joint training approach. The architecture consists of a bottom layer—either a standard RNN or an LSTM—that generates a fixed-length vector representation of the segment. On top of this shared layer, two separate feedforward networks function as "heads": a regressor for predicting similarity scores (0-5) and a classifier for predicting relationship labels (e.g., equivalence, opposition).
By backpropagating error gradients from both tasks into the shared recurrent layer, the model is forced to learn features that are robust for both classification and regression. This joint training acts as a form of regularization, improving the generalizability of the underlying representations.
Results and Insights
The system was evaluated on diverse datasets, including image descriptions, news headlines, and student answers. A key finding was that LSTMs consistently outperformed standard RNNs, demonstrating smoother learning curves and better data fitting, particularly for the more complex labeling task.
As shown in the training curves, the system generally achieved higher accuracy in predicting similarity scores than in identifying specific relationship labels. This disparity highlights the inherent difficulty of distinguishing nuanced semantic relations like "specialization" versus "similarity," especially with limited training data. Despite these challenges, iUBC achieved the top performance in the "Answer-Students" dataset for the 2016 task, proving that a neural-first approach can be highly competitive without extensive manual feature engineering.
Conclusion
The iUBC system demonstrates the effectiveness of joint neural architectures for interpretable STS. By leveraging shared representations for classification and regression, the model provides both a similarity score and an explanatory label. Future work may involve incorporating lexicalized features to improve label prediction accuracy and exploring larger datasets to further refine the recurrent layers.
Summary updated