Abstract
Estimating semantic relatedness presents a unique challenge in Natural Language Processing, as it encompasses a broader range of associations than simple semantic similarity. This paper details the Tübingen-CL team's participation in SemEval-2024 Task 1, where they developed a robust ensemble system combining surface-level text statistics with deep learning outputs. By aggregating diverse linguistic perspectives, the system achieved a top-tier Spearman correlation of 0.850 in the supervised English track and demonstrated strong performance across Spanish and Hindi in unsupervised settings.
Introduction
In Natural Language Processing (NLP), a distinction is often made between semantic similarity—where sentences share meaning or entailment—and semantic relatedness. While "teacher" and "student" are not similar in meaning, they are highly related within the educational domain. Similarly, antonyms like "tasty" and "unpalatable" are related through their shared context of food description. Most existing models are optimized for similarity; the Tübingen-CL team aimed to bridge this gap by capturing broader associations across English, Spanish, and Hindi using both supervised and unsupervised methods.
Methodology
The core hypothesis of this work was that semantic relatedness is multi-faceted and best captured by an ensemble of features. The approach integrated three primary categories:
- Textual Statistical Features: Surface-level patterns such as character-level Levenshtein distance and word overlap. A key finding was that focusing on content words (nouns, verbs, adjectives) while ignoring function words significantly improved the relatedness signal.
- Word Embedding Variations: The team experimented with static (GloVe) and contextual (BERT) embeddings, testing variations like noun-only embeddings or "Tree-Based" embeddings. Principal Component Analysis (PCA) was applied to capture significant data variations.
- Sentence-Level Representations: High-performing pre-trained models, including Sentence-BERT (mpnet-base), Jina Embedding, and the T5 large language model, were integrated into the final system.
For the Supervised Track, these features were fed into a Support Vector Machine (SVM) regression model with a Radial Basis Function (RBF) kernel. The Unsupervised Track utilized a simpler arithmetic mean of scores from various models and statistical ratios.
Key Insights and Ideas
Several critical observations emerged from the experiments:
- Content is King: Filtering out functional words before calculating overlap or embeddings consistently boosted performance. This suggests that relatedness is driven more by the "actors" and "actions" in a sentence than by grammatical "glue."
- The Power of PCA: Applying PCA to word embeddings improved their individual correlation with human judgments. As seen in the results for GloVe and BERT variations, PCA-transformed embeddings often outperformed their raw counterparts.
- Structural Limits: Interestingly, focusing on the sentence "root" (the main predicate) via dependency trees did not yield better results. Relatedness appears to be a holistic property of the entire sentence rather than just its core grammatical structure.
Key Results
The ensemble approach proved highly effective across different languages and tracks. In the supervised English track, the ensemble reached a Spearman correlation of 0.850, outperforming the best individual model (T5 base at 0.825).
As shown in the table above, the unsupervised system also achieved strong results, with correlations of 0.837 for English and 0.705 for Spanish. The improvement of the ensemble over individual models like T5 base is statistically significant in the context of competitive NLP benchmarks, validating the strategy of combining statistical "surface" features with deep learning "intuition."
Limitations and Future Work
A notable limitation was language bias, with performance on Hindi (0.649) lagging behind English and Spanish. This is likely due to the English-centric nature of the pre-training data for many underlying models. Future work will focus on low-resource languages and developing data collection strategies tailored to non-Western languages to ensure more equitable semantic modeling across the globe.
Summary updated