TL;DR
This paper introduces ILSUM, the first shared task focused on automatic text summarization for Indian languages (Hindi, Gujarati, and Indian English). It addresses the significant gap in resources for these languages compared to English, presenting a new benchmark dataset and analyzing the approaches and challenges faced by participants, highlighting the dominance of pre-trained models.
What Problem Does This Paper Address?
The field of Natural Language Processing (NLP) has historically been dominated by research on resource-rich languages like English, Spanish, and Chinese. Languages with millions of speakers, such as Hindi and Gujarati, have been largely neglected, suffering from a severe lack of large-scale, publicly available datasets and benchmarks. This resource scarcity hinders meaningful research and development in NLP tasks for these languages. While initiatives like the Forum for Information Retrieval Evaluation (FIRE) are bridging this gap, automatic text summarization, a crucial NLP task, has been particularly underserved for Indian languages. Existing datasets are often small, not publicly accessible, or too limited in scope to support the training of modern deep learning models.
Methods
The ILSUM shared task itself defined the primary method: participants were tasked with generating summaries for news articles. The task allowed for both extractive and abstractive summarization. A key methodological decision was to not enforce a fixed-length summary, instead setting a maximum length of 75 words and expecting participants to predict an appropriate length. Evaluation was conducted using standard ROUGE metrics (ROUGE-1, ROUGE-2, and ROUGE-4 F-scores).
Regarding participant approaches, the paper highlights that the vast majority of teams leveraged large pre-trained transformer models such as BART, Pegasus, and MT5. A smaller number of teams explored traditional unsupervised methods like TexRank. Notably, very few teams utilized language-specific resources like stemmers or stopword lists, likely due to their limited availability.
Datasets
The paper introduces the ILSUM dataset, a new benchmark for Indian language summarization. It comprises:
- Hindi: 10,000+ news articles
- Gujarati: 12,000+ news articles
- Indian English: 17,900+ news articles
Each article is paired with its title and headlines. A unique challenge posed by this dataset is the prevalence of code-mixing and script-mixing, where English phrases are often embedded within Hindi or Gujarati articles and headlines. The dataset was split into training (70%), testing (25%), and validation (5%) sets.
Key Insights and Ideas
- Resource Scarcity is a Major Bottleneck: The paper strongly emphasizes the critical lack of resources for Indian languages, which directly impacts research progress. The ILSUM dataset is a significant step towards alleviating this.
- Pre-trained Models Dominate: The overwhelming reliance on large pre-trained models (BART, Pegasus, MT5) by participants indicates their current dominance in tackling summarization tasks, even for low-resource languages.
- Code-Mixing is a Unique Challenge: The presence of code-mixed text in the dataset presents a distinct challenge for summarization systems, requiring them to handle linguistic variations beyond single-script text.
- No Fixed-Length Summary is a Practical Approach: Moving away from fixed-length summaries for diverse article lengths is a sensible adaptation, acknowledging the limitations of rigid constraints in real-world scenarios.
- Limited Use of Linguistic Resources: The lack of utilization of language-specific resources (stemmers, stopword lists) by participants suggests a need for better availability and evaluation of such resources for Indian languages.
Key Results
- Participation: 56 teams registered, 12 submitted runs, and 10 submitted working notes.
- Performance: The MT-NLP IIIT-H team achieved the best performance across all three languages.
- Hindi: The top F1 score for ROUGE-1 was 0.607.
- Gujarati: The top F1 score for ROUGE-1 was 0.261.
- English: The top F1 score for ROUGE-1 was 0.558.
- Comparison: The paper notes that ROUGE F-scores were comparable between English and Hindi corpora but significantly lower for Gujarati, suggesting greater difficulty in summarizing Gujarati text or a less mature state of models for this language.
- Qualitative Observations: Summaries generated by participating teams, even those that were code-mixed, were found to be comparable in quality to single-script summaries.
Limitations and Future Work
- Limitations:
- The paper acknowledges that the ROUGE scores for Gujarati were significantly lower than for Hindi and English, indicating a challenge in achieving high performance for this language.
- The limited use of language-specific linguistic resources by participants highlights a gap in their availability and integration.
- Future Work:
- The authors plan to create similar corpora for other Indian languages, including Bengali and Dravidian languages like Tamil and Telugu.
- They aim to make linguistic resources for Indian languages more accessible and encourage their use in future iterations of the task.
- The findings suggest a need for further research into handling code-mixed text effectively in summarization.
Summary updated