TL;DR
This paper tackles the challenging task of classifying narratives in news articles across multiple languages and topics using a zero-shot approach with Large Language Models (LLMs). By leveraging LLMs without task-specific training and employing machine translation to English, the UNEDTeam achieved competitive results, highlighting the potential of LLMs for multilingual narrative analysis while also revealing areas for improvement.
What Problem Does This Paper Address?
Identifying and classifying narratives within news texts is crucial for understanding discourse, detecting bias, and grasping socio-political dynamics. SemEval-2025 Task 10 specifically aims to advance this field by focusing on "Multilingual Characterization and Extraction of Narratives from Online News." This task is particularly challenging due to:
- Linguistic Diversity: News is presented in five different languages (English, Bulgarian, Portuguese, Hindi, and Russian), some of which are less represented in LLM training data.
- Subjectivity: Classifying narratives can be inherently subjective.
- Limited Labeled Data: Obtaining large, human-annotated datasets for training in multiple languages is difficult.
The task is structured into three subtasks, with this paper focusing on Subtask 2: Narrative Classification, which involves assigning relevant sub-narrative labels from a hierarchical taxonomy to news articles within specific domains (Climate Change and the Russia-Ukraine War).
Methods
The UNEDTeam adopted a Zero-Shot approach, meaning they utilized a pre-trained LLM without any fine-tuning on the specific task dataset. This strategy capitalizes on the LLM's general knowledge and ability to generalize. To address the linguistic challenges, they implemented a crucial pre-processing step: machine translation of all news articles into English using OPUS-MT models. This ensures a more homogeneous input for the LLM, especially given that LLMs often perform better in English.
The classification process was hierarchical:
- Topic Classification: The LLM first determined the main topic of the news item (either "Russia-Ukraine War" or "Climate Change").
- Sub-narrative Classification: Within the identified topic, the LLM then classified the specific sub-narratives present in the article.
The system used the calme-2.4-rys-78b model, a fine-tuned version of Qwen 2 78B, known for its strong performance. To optimize for memory and speed, the model was quantized to 4 bits. The output was requested in JSON format, including a reasoning for the classification, to enhance transparency.
Datasets
The paper utilizes the SemEval-2025 Task 10 Subtask 2 dataset. This dataset comprises news articles in five languages, covering two main thematic domains: Climate Change (CC) and the Russia-Ukraine War (URW). The dataset is structured hierarchically with main topics, main narratives, and sub-narratives. The training + development set contains 576 news items, with specific counts per language detailed in Table 1. The test set is not explicitly described in terms of size but is used for evaluation.
Key Insights and Ideas
- LLMs as Zero-Shot Learners: The study demonstrates that LLMs can be effectively employed in a zero-shot setting for complex multilingual NLP tasks like narrative classification, reducing the need for extensive task-specific labeled data.
- Machine Translation as a Bridge: Translating to English proved to be a viable strategy to overcome language-specific limitations of LLMs and enable a more unified classification process.
- Hierarchical Classification is Key: Breaking down the classification into topic and then sub-narrative levels allows for a more structured and potentially more accurate analysis.
- Sub-narrative Classification is Harder: The results indicate that while topic classification is relatively straightforward for the LLM, identifying specific sub-narratives presents a greater challenge, leading to lower performance metrics.
- Language Variability Matters: Performance varies significantly across languages, with some languages showing much lower F1 scores and higher standard deviations, suggesting that LLM generalization is not uniform.
Key Results
The UNEDTeam's zero-shot system showed variable performance across languages and tasks.
- Topic Classification: The model achieved high F1 scores (above 87%) for topic classification across all languages on the development set, indicating strong performance in distinguishing between the two main themes.
- Sub-narrative Classification (F1 Samples): Performance here was more moderate. The best performing language was Bulgarian (BG) with an F1 of 0.4248, while Hindi (HI) and Portuguese (PT) had the lowest scores (0.2305 and 0.2257, respectively) on the development set.
- Test Set Performance: A general decrease in metrics was observed on the test set compared to the development set.
- Comparison with Best Model: The system generally performed worse than the top-ranked models on the task, with notable differences in F1 Coarse (narrative classification) and F1 Samples (sub-narrative classification), particularly in Russian (RU) and Portuguese (PT). For instance, in Russian, the model achieved an F1 Coarse of 0.513 compared to the best model's 0.709.
Limitations and Future Work
The authors acknowledge several limitations:
- Model Quantization: Quantizing the model to 4 bits for efficiency might have impacted accuracy by losing information in the weights.
- Inference Time: The high inference time hinders scalability for large-scale applications.
- Machine Translation Noise: The use of machine translation can introduce noise into the text, potentially affecting classification accuracy.
- Unsupervised Zero-Shot: The lack of task-specific tuning limits the model's ability to learn finer patterns.
- Context Length: The large number of sub-narratives and the length of news stories pose challenges for LLMs in managing context.
- Single Model Run: For a more robust evaluation, multiple runs and ensemble techniques are recommended to reduce prediction variability.
Future work directions include:
- Analyzing Machine Translation Impact: Quantifying the errors introduced by machine translation.
- Direct Classification: Exploring direct classification in languages with sufficient LLM coverage.
- Ensemble Methods: Employing multiple runs and ensemble techniques for more reliable performance estimates.
- Supervised Approaches: Investigating Few-Shot or Fine-Tuning approaches using the provided training data.
- Inter-Annotator Agreement: Analyzing inter-annotator agreement on the dataset to better contextualize model performance and understand task difficulty.
Summary updated