UrduMASD: A Multimodal Abstractive Summarization Dataset for Urdu

Ali Faheem, Faizad Ullah, Muhammad Sohaib Ayub, Asim KarimCOLING 2024

TL;DR

This paper introduces UrduMASD, the first large-scale dataset for multimodal abstractive summarization of Urdu videos. By combining video, audio transcripts, and titles, this dataset aims to bridge the gap in low-resource language research and enable more effective summarization of multimedia content. Experiments show that incorporating visual information significantly improves summarization quality, even with current limitations in Urdu language models.

What Problem Does This Paper Address?

In today's multimedia-rich digital landscape, users consume vast amounts of information through videos, social media, and other multimodal content. Effectively summarizing this content is crucial for information retrieval, searchability, and comprehension. However, existing research and datasets for abstractive summarization, particularly for low-resource languages like Urdu, are scarce. Traditional text-based summarization struggles to capture the nuances present in multimodal data (audio cues, visual elements), leading to inadequate summaries. This paper addresses this gap by creating a dedicated dataset for Urdu multimodal abstractive summarization, enabling research into more comprehensive and context-aware summarization systems.

Methods

The core of this work is the creation of the UrduMASD dataset. The authors collected videos and their associated titles and descriptions from Urdu news channels on YouTube. They then generated audio transcriptions using an Automatic Speech Recognition (ASR) model for Urdu. The human-written descriptions were used as the target summaries. To ensure data quality, they systematically cleaned the descriptions by removing irrelevant links and hashtags. For transcriptions, they addressed potential issues like hallucination (repeated text) by removing redundant patterns and handled mixed-language content by retaining traces of English but removing predominantly non-Urdu segments.

For evaluation, they employed intrinsic metrics:

  • Abstractivity: Measures how much the summary deviates from the original text.
  • Compression: Quantifies the reduction in content.
  • Redundancy: Assesses the repetition of information.
  • Semantic Coherence: Evaluates the logical flow between consecutive sentences in the summary.

They also experimented with two main model architectures:

  • mT5: A multilingual text-to-text transformer model, fine-tuned for summarization.
  • MLASK: An architecture that combines text and video features using multi-head attention, initialized with mT5 weights.

Datasets

  • UrduMASD: The primary contribution of this paper. It comprises 15,374 video-audio-text collections with corresponding abstractive text summaries in Urdu.
  • How2: An English multimodal summarization dataset used for comparison during intrinsic evaluation.

Key Insights and Ideas

  • The critical role of visual information: Even with current limitations in Urdu language models, incorporating visual data (either through image captions or direct video features) significantly boosts summarization performance.
  • Low-resource language challenges: The semantic coherence scores for Urdu were lower than for English datasets like How2, highlighting the need for better Urdu-specific language models and pre-training strategies.
  • Dataset quality matters: The UrduMASD dataset exhibits high abstractivity and low redundancy, making it a strong benchmark for abstractive summarization tasks.
  • Text-based features still strong: While multimodal fusion shows promise, current models perform better when relying primarily on text-based features (transcripts and image captions) for Urdu summarization, suggesting that complex fusion mechanisms might need further refinement.

Key Results

  • The UrduMASD dataset achieved high abstractivity (0.958) and low redundancy (0.111) scores, comparable to or better than the English How2 dataset.
  • Incorporating visual information (image captions) with text transcripts led to an improvement in ROUGE scores by 2.6% compared to using only text transcripts.
  • The MLASK model, which uses multimodal fusion, achieved competitive ROUGE scores, though slightly lower than text-only approaches with mT5.
  • The average summary length in UrduMASD (50.92 words) is longer than in some English benchmarks, suggesting the generation of more comprehensive summaries.

Limitations and Future Work

  • Limitations:
    • The performance of out-of-the-box multilingual models (mT5) for Urdu is a concern, suggesting a need for dedicated Urdu pre-training.
  • Computational constraints limited the exploration of various mT5 variants.
  • The MLASK model was trained for a limited number of epochs, potentially limiting its full potential.
  • The semantic coherence scores were lower due to limitations in the multilingual BERT model's comprehension of Urdu.
  • Future Work:
    • Pre-training mT5 specifically for the Urdu generation task.
  • Exploring more advanced techniques for multimodal feature fusion.
  • Further investigation into improving the semantic coherence of summaries for Urdu.
  • Making the dataset available for research purposes.

Summary updated

BibTeX
@inproceedings{faheem2024urdumasd,
  author = {Ali Faheem and Faizad Ullah and Muhammad Sohaib Ayub and Asim Karim},
  title = {{UrduMASD: A Multimodal Abstractive Summarization Dataset for Urdu}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure