TL;DR
This paper introduces a novel self-instruction learning method to improve Large Language Models' (LLMs) ability to distinguish Semantic Textual Relatedness (STR) from Semantic Textual Similarity (STS). By iteratively refining task instructions based on human feedback and LLM-generated scores, their system significantly boosts performance on multilingual STR tasks, outperforming conventional models.
What Problem Does This Paper Address?
The paper tackles a subtle but important challenge in Natural Language Understanding: differentiating between Semantic Textual Relatedness (STR) and Semantic Textual Similarity (STS). While STS focuses on how similar two sentences are in meaning, STR goes deeper, evaluating the logical connection and coherence between them. This distinction is crucial for tasks like question answering or dialogue systems, where understanding the relationship between statements is paramount. However, many powerful LLMs, often trained on STS datasets, struggle to make this fine-grained distinction. This can lead to misinterpretations, especially in cases where sentences are superficially similar (high STS) but lack a true logical connection (low STR). The SemEval-2024 Task 1 specifically highlights this challenge, requiring models to excel at STR across various languages.
Methods
The core innovation of this paper is a self-instruction learning framework designed to enhance LLMs' STR capabilities. The process begins with a basic task description. This description is then fed into an LLM, which generates an initial "task instruction." This instruction is refined through a feedback loop:
- Human Feedback & LLM Generation: The system uses human feedback to guide the LLM in generating new, improved task instructions.
- Iterative Refinement: The LLM then iteratively refines these instructions by comparing them to the original and evaluating the resulting scores. This process aims to create instructions that better capture the nuances of STR.
- Black-box Optimization: The LLM itself acts as a "black-box optimizer," using the generated scores to progressively optimize the instructions.
- Back Translation: For low-resource languages, sentence pairs are translated into English using back-translation to serve as input for the LLMs.
The authors also explored different instruction design strategies, including adding "chain-of-thought" reasoning and "easily confused examples" to the instructions, finding that combining these elements significantly improved performance.
Datasets
The research utilizes datasets for SemEval-2024 Task 1, which includes sentence pairs in 14 distinct languages. Specifically, 9 of these languages were used for Track A. The paper does not introduce new datasets but focuses on improving performance on existing multilingual STR benchmarks.
Key Insights and Ideas
- STR vs. STS is a real challenge for LLMs: Even powerful models struggle to differentiate these two concepts, leading to performance gaps.
- Self-instruction is a powerful tool for fine-tuning LLMs: Iteratively refining task instructions, guided by feedback and performance scores, can significantly improve an LLM's ability to handle nuanced tasks.
- Combining instruction elements matters: Simply providing a task description isn't enough. Incorporating chain-of-thought and examples of tricky cases within the instructions helps LLMs reason more effectively.
- Back-translation is a viable, though imperfect, solution for low-resource languages: It enables the use of LLMs but can be affected by language biases.
Key Results
The proposed self-instruction method, when applied to the ERNIE-Bot-turbo LLM, achieved a score enhancement of 4 to 7% on multilingual development datasets for Track A compared to the non-optimized approach. This improvement demonstrates the effectiveness of their iterative instruction refinement strategy. The system also achieved commendable rankings in the final test evaluation across various languages.
Limitations and Future Work
The authors acknowledge that the back-translation method can encounter challenges with low-resource languages, particularly Arabic, due to language biases. Additionally, they note that the scoring model (ERNIE-Bot-turbo), trained primarily on Chinese and English corpora, might exhibit weaker performance on English sentence pairs. Future work aims to further refine LLMs' capacity to grasp the overall semantic meaning of sentences, suggesting continued exploration into improving their understanding of complex linguistic relationships.
Summary updated