Key Takeaways from the Second Shared Task on Indian Language Summarization (ILSUM 2023)

Shrey Satapara, Parth Mehta, Sandip Modha, Debasis GangulyFIRE (Working Notes) 2023 · OpenReview

TL;DR

The ILSUM 2023 shared task expanded its focus on Indian language summarization by introducing Bengali and a new subtask for detecting factual inaccuracies in machine-generated summaries. This work highlights the continued reliance on pre-trained large language models while also pointing to promising avenues for improvement, particularly in leveraging linguistic features like Named Entity Recognition.

What Problem Does This Paper Address?

The field of Natural Language Processing (NLP) has seen tremendous progress, but this advancement has largely been concentrated in resource-rich languages like English. Indian languages, despite their vast number of speakers, often lag behind due to a scarcity of high-quality datasets and research benchmarks. The Indian Language Summarization (ILSUM) initiative aims to bridge this gap by creating standardized evaluation platforms and datasets for automatic text summarization in various Indian languages. This year's ILSUM 2023 builds upon previous efforts by expanding language coverage and introducing a critical new challenge: ensuring the factual accuracy of summaries generated by AI, especially in an era of increasingly sophisticated language models that could be misused to spread misinformation.

Methods

The ILSUM 2023 task involved two main components:

  • Text Summarization: Participants were tasked with generating summaries (either extractive or abstractive) for articles in Hindi, Gujarati, Bengali, and Indian English. The primary evaluation metrics used were standard ROUGE scores and BERTScore, which offers a more nuanced evaluation of abstractive summaries.
  • Factual Incorrectness Detection: This novel subtask focused on identifying misinformation within machine-generated summaries. Participants were given summaries and had to classify them into categories such as misrepresentation, inaccurate quantities, false attribution, or fabrication, alongside a "correct" category. The dataset for this subtask was generated using GPT-4 for incorrect examples and GPT-3.5 for correct ones. Evaluation was performed using Macro F1 score.

A notable observation was the widespread use of pre-trained large language models (LLMs) like mT5, MBART, and IndicBART by participating teams for the summarization task. One team, however, showed promise by combining Named Entity Recognition (NER) with pre-trained LLMs, suggesting that incorporating linguistic knowledge can enhance performance.

Datasets

ILSUM 2023 introduced and expanded upon several datasets:

  • Indian Language Summarization Dataset: This year's dataset was significantly expanded, featuring approximately 20,000 article-summary pairs across Hindi, Gujarati, Bengali, and Indian English. This builds upon the previous edition by adding Bengali and increasing the number of articles for existing languages. The dataset also incorporates challenges like code-mixing and script mixing, common in real-world Indian language content.
  • Factual Incorrectness Detection Dataset: A new benchmark dataset was created for this subtask, containing machine-generated summaries labeled with one of five categories: misrepresentation, inaccurate quantities, false attribution, fabrication, or correct.

Key Insights and Ideas

  • LLMs Dominate, but Linguistic Features Matter: While pre-trained LLMs are the go-to for summarization, the success of the IIT-BHU team in Hindi, which incorporated NER, suggests that hybrid approaches can offer competitive or even superior performance. This highlights the potential for combining the power of LLMs with traditional linguistic insights.
  • The Growing Threat of AI-Generated Misinformation: The introduction of the misinformation detection subtask underscores the urgent need to develop robust methods for verifying the accuracy of AI-generated content. This is particularly critical for summaries, where subtle alterations can significantly distort the original meaning.
  • Code-Mixing is a Persistent Challenge: The continued presence of code-mixing (mixing of languages within a single text) in the datasets presents a unique challenge for summarization systems, requiring models that can effectively handle multilingual inputs.

Key Results

  • Summarization Performance: While specific quantitative results varied by language and team, the paper indicates that teams using LLMs generally performed well. The Irlab-IITBHU team achieved top performance in Hindi summarization, leveraging NER. For Gujarati and Bengali, a single submission from BITS Pilani using mT5 was the sole entry. In English summarization, NITK - AI secured the top spot.
  • Misinformation Detection: Only one team participated in the misinformation detection subtask, utilizing GPT-3.5 Turbo with zero-shot prompting. They achieved a Macro F1 score of 0.527, indicating room for improvement in this nascent area.

Limitations and Future Work

The authors acknowledge that the misinformation detection subtask is still in its early stages, with only one participating team. For future work, they plan to:

  • Expand Language Coverage: Extend the summarization subtask to include more Indian languages, particularly Dravidian languages.
  • Refine Misinformation Detection: Move towards providing more granular annotations for factual inaccuracies, pinpointing specific parts of a summary that are incorrect rather than just labeling the entire summary. This will enable more precise model training and evaluation.
  • Explore Hybrid Approaches: Further investigate the benefits of combining LLMs with linguistic features for improved summarization performance.

Summary updated

BibTeX
@inproceedings{satapara2023key,
  author = {Shrey Satapara and Parth Mehta and Sandip Modha and Debasis Ganguly},
  title = {{Key Takeaways from the Second Shared Task on Indian Language Summarization (ILSUM 2023)}},
  booktitle = {FIRE (Working Notes)},
  year = {2023},
  url = {https://openreview.net/forum?id=KtvpVTvAFW},
}
Figure