TL;DR
This paper introduces HeSum, the first large-scale dataset for abstractive text summarization in Hebrew, addressing a critical gap in evaluating large language models (LLMs) for morphologically rich languages. The authors demonstrate that HeSum presents unique challenges for current LLMs, highlighting the need for specialized resources and evaluation methods for languages beyond English.
What Problem Does This Paper Address?
While large language models (LLMs) have shown remarkable capabilities in English for various natural language tasks, their performance in lower-resourced languages, particularly for generative tasks like abstractive summarization, remains largely unknown. Hebrew, a morphologically rich language (MRL), presents additional complexities due to word ambiguity and intricate meaning construction, making it a particularly challenging testbed. Existing evaluations are heavily skewed towards English, leaving a significant gap in our understanding of LLM capabilities for MRLs. This paper aims to bridge this gap by introducing a novel dataset and evaluation framework for Hebrew abstractive text summarization.
Methods
The core of this work is the creation of the HeSum dataset, a collection of 10,000 article-summary pairs sourced from professional Hebrew news websites. The authors conducted a thorough linguistic analysis of the dataset, confirming its high abstractness and identifying unique morphological challenges specific to Hebrew. They then evaluated several state-of-the-art LLMs, including GPT-4, GPT-3.5, and a fine-tuned mLongT5 model, using both automatic metrics (ROUGE, BertScore) and human evaluation. The experimental setup involved exploring different prompting strategies, including translating prompts to English to leverage LLMs' stronger performance in that language.
Datasets
- HeSum: A newly introduced dataset comprising 10,000 article-summary pairs from Hebrew news websites. It is designed to specifically address the challenges of abstractive text summarization in Modern Hebrew, featuring high abstractness and unique morphological complexities.
- English Benchmarks (for comparison): CNN/Daily Mail and XSum datasets were used to provide a comparative analysis of HeSum's abstractness and challenges against established English summarization datasets.
Key Insights and Ideas
- Hebrew's Morphological Richness is a Major Hurdle: The paper strongly emphasizes that the inherent complexity of Hebrew's morphology significantly complicates text understanding and generation for LLMs, posing challenges not present in less morphologically rich languages like English.
- Abstractness is Key, and HeSum Delivers: HeSum summaries are demonstrably more abstractive than those in English benchmarks, with a substantial portion of unique vocabulary and bigrams not appearing in the original articles. This makes it a robust test for generative capabilities.
- ROUGE Isn't Enough for MRLs: The study found a negative correlation between ROUGE scores and human evaluations for Hebrew summarization. This highlights the limitations of n-gram overlap metrics when dealing with languages that allow for flexible word order and rich morphology, suggesting a need for more semantically aware evaluation metrics.
- GPT Models Outperform Fine-tuned Models in Nuance: While fine-tuned models like mLongT5 achieved higher ROUGE scores, the GPT-based models demonstrated superior performance in BertScore (semantic similarity) and human evaluations for coherence and completeness, suggesting they better capture the nuances of Hebrew summarization.
Key Results
- HeSum summaries exhibit high abstractness, with approximately half of their unique vocabulary and over 73% of bigrams being novel compared to the original articles.
- HeSum presents a significant compression challenge, with summaries averaging less than 5% of the input article length and minimal redundancy.
- In automatic evaluations, mLongT5 achieved higher ROUGE scores, while GPT-4 and GPT-3.5 performed better on BertScore.
- Human evaluation revealed that GPT-based models were rated higher for coherence and completeness compared to the fine-tuned mLongT5.
- A qualitative error analysis showed that while mLongT5 struggled with repetition and exact copying, GPT-based models made errors related to Hebrew-specific morphological phenomena like incorrect gender and definiteness marking.
Limitations and Future Work
- Limited LLM Scope: The current analysis is limited to two generative models (GPT-4 and GPT-3.5) and one fine-tuned model (mLongT5), potentially overlooking newer models with superior performance.
- Domain Specificity: HeSum primarily consists of news articles, which might bias LLM performance towards news-style writing and not fully capture linguistic diversity across other genres.
- Evaluation Metric Limitations: The paper reiterates the limitations of n-gram-based metrics like ROUGE for evaluating Hebrew summarization and calls for further research into developing metrics that better correlate with human judgment.
- Future Work: The authors suggest expanding the evaluation to include few-shot settings and a broader range of LLMs. They also aim to explore HeSum's potential for other generative tasks in MRLs and encourage the development of more robust evaluation metrics.
Summary updated