WordWiz at SemEval-2025 Task 10: Optimizing Narrative Extraction in Multilingual News via Fine-Tuned Language Models

Ruhollah Ahmadi, Hossein ZeinaliACL 2025

TL;DR

The WordWiz team tackled the challenge of automatically extracting concise explanations for dominant narratives in multilingual news. They achieved top rankings by combining advanced text preprocessing with fine-tuned large language models (LLMs) using both supervised fine-tuning (SFT) and direct preference optimization (DPO). This work demonstrates that sophisticated LLM fine-tuning and inference strategies can significantly improve narrative extraction performance across diverse languages.

What Problem Does This Paper Address?

In today's information-saturated world, discerning the underlying narratives and potential biases in news is crucial, especially during crises where disinformation thrives. The SemEval 2025 Task 10 specifically addresses this by focusing on "Multilingual Characterization and Extraction of Narratives from Online News." The paper hones in on Subtask 3: Narrative Extraction, which requires generating a short, evidence-based explanation (under 80 words) that supports a given dominant narrative identified in a news article. This is a complex text-to-text generation task that demands not only identifying relevant textual evidence but also synthesizing it into a coherent and concise summary that aligns with the narrative. The challenge is amplified by the multilingual nature of the task, spanning five languages: English, Bulgarian, Hindi, Portuguese, and Russian.

Methods

The WordWiz team employed a two-pronged approach to tackle narrative extraction, leveraging state-of-the-art large language models (LLMs):

  • Supervised Fine-Tuning (SFT): They fine-tuned several pre-trained LLMs (Phi-3.5, Llama-3, Llama-3.1-8B) on the task's training data. This involved structuring prompts to clearly define task requirements, including input documents, narrative information, and desired output format. Language-specific tokens were also incorporated to aid multilingual performance.
  • Direct Preference Optimization (DPO): To further refine the models, they utilized DPO. This technique allows LLMs to learn from human preferences between pairs of generated explanations, aligning outputs more closely with human judgments of quality and relevance without needing an explicit reward model. For their DPO implementation, they used the SFT-tuned Phi-3.5 model as a reference and trained it using human-annotated explanations as preferred responses and outputs from a non-fine-tuned model as rejected responses.

Additionally, they developed a robust preprocessing pipeline to enhance data quality. This pipeline involved removing noise (emojis, URLs, hashtags), standardizing text (punctuation, whitespace, case normalization), enriching semantic context (expanding abbreviations), and filtering redundant sentences.

For inference, they implemented a multi-candidate generation strategy. The system generated three candidate explanations for each article-narrative pair using varying temperature parameters (0.5, 0.7, 0.9). The best explanation was then selected based on a weighted score combining keyword overlap (matching narrative-specific keywords from the text) and length scoring (favoring explanations close to the 80-word limit).

Datasets

The primary dataset used was the official SemEval 2025 Task 10 dataset. This dataset consists of news articles annotated with dominant narratives and, where applicable, sub-narrative classifications. It includes gold-standard explanations for training and evaluation. The dataset covers five languages (English, Bulgarian, Hindi, Portuguese, Russian) and two domains: climate change (CC) and the Ukraine-Russia war (UA). The authors used the development set for experimental configuration assessment and the training and development sets for final model fine-tuning.

Key Insights and Ideas

  • Smaller models can outperform larger ones: Surprisingly, the smaller Phi-3.5 model consistently outperformed the larger Llama-3.1-8B across most languages, suggesting that model architecture and fine-tuning strategies are more critical than sheer parameter count for this task.
  • DPO offers nuanced explanations but can reduce recall: While DPO improved precision and generated more comprehensive, contextually enriched explanations, it led to a slight decrease in recall compared to SFT. This indicates a trade-off between selectivity and coverage.
  • Preprocessing is paramount: The advanced preprocessing pipeline, which cleaned and enriched the data, was a key factor in improving model performance and ensuring cross-lingual consistency.
  • Multi-temperature inference is effective: Generating multiple candidates with varying temperatures and selecting the best one based on relevance significantly boosted performance over single-temperature strategies.

Key Results

The WordWiz system achieved remarkable success in the SemEval 2025 Task 10:

  • First place in Portuguese and second place in English, Russian, Bulgarian, and Hindi.
  • The top-performing model, Phi-3.5, consistently exceeded baseline performance across all languages, with notable improvements in English (+8.05%) and Russian (+6.98%).
  • The system demonstrated strong performance in Portuguese and English, with the lowest improvement observed in Bulgarian (+4.96%).
  • In English, the DPO-tuned Phi-3.5 model showed higher precision (+0.73%) than SFT but a lower F1 macro score due to reduced recall.

Limitations and Future Work

The authors acknowledge that while DPO improved explanation quality, it came at the cost of slightly lower recall. Future research directions suggested include:

  • Extracting evidence from news text: Incorporating more advanced techniques for identifying and extracting the specific textual evidence that supports the narrative.
  • Expanding multilingual capabilities: Investigating cross-lingual transfer learning and extending the system's applicability to low-resource languages.
  • Exploring more advanced evidence extraction: Developing techniques that can further ground explanations in the source material.

Summary updated

BibTeX
@inproceedings{ahmadi2025wordwiz,
  author = {Ruhollah Ahmadi and Hossein Zeinali},
  title = {{WordWiz at SemEval-2025 Task 10: Optimizing Narrative Extraction in Multilingual News via Fine-Tuned Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure