TL;DR
This paper tackles the Semantic Textual Relatedness (STR) task, aiming to quantify how semantically similar two sentences are. The WarwickNLP team achieved strong results by employing a cross-encoder architecture enhanced with a parameter-efficient Low-Rank Adapter (LoRA), demonstrating its effectiveness for both English and Moroccan Arabic.
What Problem Does This Paper Address?
Semantic Textual Relatedness (STR) is a fundamental task in Natural Language Processing (NLP) that aims to measure the degree of semantic connection between two pieces of text. This is crucial for various downstream applications like information retrieval, question answering, and text summarization. While previous work has focused on Semantic Textual Similarity (STS), STR is a more general concept, encompassing broader semantic connections beyond mere similarity. The SemEval-2024 Task 1 specifically targets STR by asking participants to predict a score between 0 and 1 indicating the relatedness of sentence pairs across multiple languages. The challenge lies in developing models that can accurately capture these nuanced semantic relationships efficiently, especially for languages with limited training data.
Methods
The WarwickNLP team's approach centers on a cross-encoder architecture. Unlike bi-encoders that process sentences independently and then compare their representations, cross-encoders process both sentences simultaneously, allowing for richer interaction modeling through cross-attention mechanisms. This architecture is known for its effectiveness in capturing fine-grained semantic relationships.
To address the challenge of efficient fine-tuning, especially for languages with smaller datasets like Moroccan Arabic, they incorporated Parameter-Efficient Fine-Tuning (PEFT) techniques. Specifically, they utilized Low-Rank Adapters (LoRA). Instead of fine-tuning the entire pre-trained model, LoRA injects small, trainable low-rank matrices into specific layers of the transformer architecture. This significantly reduces the number of trainable parameters, making the fine-tuning process faster and more memory-efficient, while aiming to maintain high accuracy.
Their system involves initializing a LoRA adapter with random weights, training it on the provided datasets, and then merging it with the classification head of a pre-trained cross-encoder. This combined model is then used to predict the relatedness score.
Datasets
The primary datasets used in this work are from the SemEval-2024 Task 1 on Semantic Textual Relatedness (STR). These datasets provide pairs of sentences annotated with a human-generated score between 0 and 1, indicating their relatedness. The task organizers provided data for nine languages or dialects. The WarwickNLP team focused on English and Moroccan Arabic.
Additionally, the paper mentions the use of the STS-B dataset for pre-training the cross-encoder. The STS-B dataset is a well-established benchmark for Semantic Textual Similarity, and its use for pre-training the cross-encoder proved beneficial for the STR task.
Key Insights and Ideas
- LoRA's Efficiency for Low-Resource Languages: The paper highlights the effectiveness of LoRA for languages with limited training data, such as Moroccan Arabic. By reducing the number of trainable parameters, LoRA enables efficient fine-tuning without sacrificing significant performance, making it a valuable technique for cross-lingual NLP tasks.
- Cross-Encoder Superiority for Nuanced Relatedness: The choice of a cross-encoder architecture, which allows for direct interaction between sentence representations, proved crucial for capturing the subtle semantic relationships required for STR, outperforming simpler embedding-based approaches.
- Pre-training on STS-B Benefits STR: Pre-training the cross-encoder on the STS-B dataset, a task focused on similarity, provided a strong foundation for the STR task, leading to better performance than using general-purpose pre-trained models.
- Adapter Merging is Key: The experiments showed that merging different types of adapters (Houlsby, Pfeiffer, and LoRA) with the cross-encoder was beneficial, with LoRA ultimately yielding the best results.
Key Results
The WarwickNLP system achieved competitive results in the SemEval-2024 Task 1:
- English: Ranked eighth with a Pearson Coefficient score of 0.8425. This score exceeded the provided baseline of 0.83.
- Moroccan Arabic: Ranked seventh with a Pearson Coefficient score of 0.8163. This score also surpassed the baseline of 0.77.
During development, the LoRA adapter achieved a development score of 0.8343, which was further improved to 0.8417 after increasing training epochs.
Limitations and Future Work
The authors acknowledge several limitations:
- Partial Fine-Tuning: The models were not fully fine-tuned with the training data; only the adapters were trained. Comparing full fine-tuning to adapter tuning would provide a more comprehensive understanding.
- Lack of Hyper-parameter Optimization: Beyond the number of epochs, extensive hyper-parameter tuning was not performed. Further exploration of hyper-parameters could lead to improved performance.
- Full Cross-Encoder Training: The paper suggests that fully training the cross-encoder from scratch, rather than relying on pre-trained models, could be an interesting avenue for future research.
- No Pre-processing: The sentences were tokenized as they were, without any specific pre-processing steps. Investigating the impact of pre-processing could be beneficial.
They also noted that a CAMELBERT model, pre-trained on a large Arabic corpus, achieved a higher development score (0.871) for Moroccan Arabic than their chosen cross-encoder approach, but this was not submitted for official results due to consistency reasons. This suggests that domain-specific pre-training can be highly advantageous.
Summary updated