TL;DR
This paper introduces two new semantic datasets for Maltese, a low-resource language, by leveraging existing news corpora and their metadata. These datasets, designed for topic classification and headline generation, aim to push the boundaries of Maltese NLP beyond basic syntactic tasks and reveal current limitations in existing models' semantic understanding.
What Problem Does This Paper Address?
The development of Natural Language Processing (NLP) tools for low-resource languages like Maltese is significantly hampered by a lack of readily available, high-quality datasets. While unsupervised pre-training has advanced NLP, labeled datasets are still crucial for evaluating model performance. Existing datasets for low-resource languages often focus on low-level syntactic tasks (like part-of-speech tagging) rather than higher-level semantic understanding. This paper aims to bridge this gap by creating new semantic benchmarks for Maltese, specifically for topic classification and abstractive summarization (headline generation), using a practical, semi-automated approach to overcome the data scarcity challenge.
Methods
The authors employed a semi-automated pipeline to construct their datasets, primarily leveraging an existing Maltese news corpus (Korpus Malti v4.0).
- Data Collection & Preprocessing: Articles were scraped from various Maltese news portals. Initial filtering removed non-article pages, normalized URLs, and excluded articles with minimal content (less than ten words). Further preprocessing involved removing CSS/JavaScript code and repetitive phrases often found in scraped web content.
- Topic Classification Dataset: This dataset was built using the tags already assigned by news portals as weak labels. To address the lack of standardization in tags across different portals, two native Maltese speakers manually merged similar tags into higher-level categories. This process involved merging tags (e.g., "EU" and "European Union" into "EU"), dropping overly specific tags, and removing articles tagged with generic categories or tags with very low article counts. Categories with high co-occurrence (>75%) were also pruned to avoid redundancy.
- Headline Generation Dataset: This dataset was created by pairing article content with its corresponding title. No additional filtering was applied beyond the initial preprocessing steps. The base URL of each article was also retained to potentially allow for style-based generation in future work.
Datasets
- Korpus Malti v4.0 (press_mt portion): The primary source corpus for both datasets. It contained approximately 44,823 articles initially.
- Maltese News Category Classification Dataset: A multi-label classification dataset derived from the news corpus. After filtering and manual tag merging, it comprises 15,374 articles assigned to 17 categories.
- Maltese News Headline Generation Dataset: A dataset of article-title pairs, consisting of 25,403 instances, derived from the news corpus without further filtering.
Key Insights and Ideas
- Leveraging Metadata for Low-Resource NLP: The paper demonstrates a practical and effective method for creating semantic datasets for low-resource languages by utilizing existing web corpora and their associated metadata (like tags and titles), significantly reducing the need for extensive manual annotation.
- Semantic Tasks are Challenging for Maltese Models: Current Maltese-specific models (BERTu, mBERTu) perform better than general multilingual models on topic classification, but their performance is notably lower than on other tasks, suggesting that semantic understanding is a more difficult challenge for Maltese.
- Multilingual Models Struggle with Abstractive Tasks: State-of-the-art multilingual decoder models (BLOOMZ, mT0) perform poorly on the headline generation task, achieving very low BLEU scores. This highlights a significant gap in their ability to perform abstractive summarization for Maltese, likely due to limited exposure to Maltese during pre-training and the inherent difficulty of the task.
- The Importance of Domain-Specific Pre-training: While not explicitly tested as a separate dataset, the results for BERTu and mBERTu suggest that pre-training on Maltese news data provides a significant advantage for tasks within that domain, even if the semantic complexity is high.
Key Results
- News Category Classification:
- Maltese-specific models (BERTu, mBERTu) outperformed general multilingual models (BERT, mBERT).
- BERTu achieved the highest macro-averaged F1 score (67.04 ± 2.02), slightly better than mBERTu (66.65 ± 1.34).
- The performance gap between Maltese-specific and general models was smaller than observed in previous studies on other Maltese tasks, indicating the task's difficulty.
- News Headline Generation:
- All models (BLOOMZ, mT0) achieved very low BLEU scores (BLOOMZ: 0.84 ± 0.36, mT0: 0.28 ± 0.14), indicating a failure to generate meaningful headlines. This is attributed to the models not being specifically fine-tuned on Maltese for this task and the inherent challenges of summarization.
Limitations and Future Work
- Dataset Size: While larger than some existing Maltese datasets, the created datasets are still relatively small compared to those available for high-resource languages.
- Model Performance on Semantic Tasks: The paper acknowledges that current models, even those pre-trained on Maltese, struggle with higher-level semantic tasks like topic classification and particularly headline generation.
- Future Work:
- Enhance existing models to improve performance on these tasks, potentially through language adaptation techniques using Maltese corpora.
- Address the data scarcity problem for low-resource languages by developing novel modeling approaches.
- Investigate the impact of retaining news portal information for future work on style-based generation.
Summary updated