WXImpactBench: A Disruptive Weather Impact Understanding Benchmark for Evaluating Large Language Models

Yongan Yu, Qingchen Hu, Xianda Du, Jiayin Wang, Fengran Mo, Renée Sieber

TL;DR

Large Language Models (LLMs) are increasingly being used to understand complex real-world phenomena, but their ability to grasp the nuanced impacts of disruptive weather events remains largely untested. This paper introduces WXImpactBench, the first benchmark designed to evaluate LLMs on this critical task, using a novel dataset derived from historical newspapers. The findings reveal that current LLMs struggle with this domain, highlighting the need for more robust models and specialized benchmarks for climate change adaptation.

What Problem Does This Paper Address?

Climate change adaptation hinges on understanding how disruptive weather events impact society. While Large Language Models (LLMs) show promise in processing vast amounts of text, their effectiveness in comprehending the complex, often subtle, impacts of weather events is largely unknown. This is primarily due to the difficulty in collecting high-quality, domain-specific data and the lack of standardized benchmarks for evaluation. Historical newspapers offer a rich source of information on how communities adapted to and recovered from disasters, but processing this data is challenging due to its non-digital format, historical linguistic variations, and potential OCR errors. This paper aims to bridge this gap by creating a dedicated dataset and benchmark for evaluating LLMs' understanding of disruptive weather impacts.

Methods

The authors developed a four-stage pipeline to construct their dataset and benchmark:

  1. Corpus Collection: Historical newspaper articles were collected from two distinct periods to capture linguistic and social shifts.
  2. Post-OCR Correction: A sophisticated process using GPT-40 was employed to correct errors in the initial OCR-digitized text, significantly improving text quality for analysis.
  3. Topic-Aware Article Selection: Latent Dirichlet Allocation (LDA) topic modeling was used to identify relevant articles, followed by manual curation by domain experts to ensure high-quality, informative samples.
  4. Manual Label Annotation: Human experts annotated the selected articles with six predefined impact categories: Infrastructural, Agricultural, Ecological, Financial, Human Health, and Political.

Based on this dataset, they proposed the WXImpactBench benchmark, which includes two evaluation tasks:

  • Multi-label Classification: Models predict the presence or absence of the six impact categories for each article.
  • Ranking-based Question Answering (QA): Models are tasked with ranking relevant articles based on their ability to answer specific questions about weather impacts.

Datasets

The paper introduces a newly constructed disruptive weather impact dataset derived from historical newspaper articles. This dataset was processed through a four-stage pipeline, including post-OCR correction and expert annotation. The benchmark, WXImpactBench, utilizes this dataset for its evaluation tasks. The dataset includes both "long context" and "mixed context" versions to evaluate LLMs' performance on different passage lengths and their ability to handle context de-noising.

Key Insights and Ideas

  • Historical Newspapers as a Rich Data Source: The study highlights the immense value of historical newspapers for understanding societal responses to weather events, offering a depth and nuance often missing in modern meteorological records.
  • The Challenge of Linguistic Evolution: The paper underscores the difficulty LLMs face in interpreting historical language, spelling variations, and evolving narrative styles, which are prevalent in older newspaper articles.
  • Polysemy in Climate Terms: The authors point out that common climate-related terms (like "blizzard" or "flood") can have multiple meanings (meteorological, metaphorical, or even as proper nouns), posing a significant disambiguation challenge for LLMs.
  • Historical Narratives are Easier for LLMs: Surprisingly, LLMs performed better on historical articles than modern ones. This is attributed to the more structured, explicit, and empirical nature of historical narratives, which makes cause-and-effect relationships clearer for models.
  • Scaling Laws Don't Guarantee Domain Understanding: While larger LLMs generally perform better, the study shows that even state-of-the-art models struggle with the specific domain of disruptive weather impacts, indicating that sheer model size isn't enough.

Key Results

  • LLMs Struggle with Disruptive Weather Impacts: Across various models and tasks, F1-scores for multi-label classification remained consistently low, particularly for political and ecological categories. Financial, agricultural, and human health impacts showed slightly better, though still suboptimal, performance.
  • Row-wise Accuracy is a Stricter Metric: This metric, requiring exact matches across all labels, revealed a significant drop in performance, emphasizing the difficulty LLMs have in correctly identifying all impacts for a given article.
  • Long-Context De-noising is Tricky: Segmenting long articles into smaller chunks improved classification accuracy in many cases, suggesting that smaller, focused contexts help LLMs. However, this benefit was less apparent in row-wise accuracy, indicating that while individual category performance might improve, overall accuracy across all labels doesn't necessarily follow.
  • One-Shot Learning Offers Limited Gains: Providing a single example in the prompt (one-shot learning) showed minimal improvement and sometimes even degraded performance, suggesting LLMs lack sufficient foundational knowledge to leverage these examples effectively for this domain.
  • Qwen2.5-14B-IT Leads in Ranking QA: This model achieved the best performance in the ranking-based question-answering task, outperforming even GPT-40.
  • Historical vs. Modern Performance: Models performed better on historical articles, likely due to their more explicit and structured narrative style.

Limitations and Future Work

  • Potential Biases: The authors acknowledge that the benchmark might have biases due to underrepresented historical events and linguistic variations.
  • Expanding Evaluation: Future work could involve evaluating a wider range of LLMs, exploring different strategies, and designing new tasks to further strengthen the benchmark.
  • Practical QA Settings: The current ranking-based QA task is a step towards real-world applications, but future research could explore more practical open-retrieval settings where models identify relevant articles from a vast database.
  • Answer Span Extraction: The current QA task focuses on ranking, leaving answer span extraction or generation for future studies.

Summary updated

Abstract

Climate change adaptation requires the understanding of disruptive weather impacts on society, where large language models (LLMs) might be applicable. However, their effectiveness is under-explored due to the difficulty of high-quality corpus collection and the lack of available benchmarks. The climate-related events stored in regional newspapers record how communities adapted and recovered from disasters. However, the processing of the original corpus is non-trivial. In this study, we first develop a disruptive weather impact dataset with a four-stage well-crafted construction pipeline. Then, we propose WXImpactBench, the first benchmark for evaluating the capacity of LLMs on disruptive weather impacts. The benchmark involves two evaluation tasks, multi-label classification and ranking-based question answering. Extensive experiments on evaluating a set of LLMs provide first-hand analysis of the challenges in developing disruptive weather impact understanding and climate change adaptation systems. The constructed dataset and the code for the evaluation framework are available to help society protect against vulnerabilities from disasters.
BibTeX
@inproceedings{yu2025wximpactbench,
  author = {Yongan Yu and Qingchen Hu and Xianda Du and Jiayin Wang and Fengran Mo and Renée Sieber},
  title = {{WXImpactBench: A Disruptive Weather Impact Understanding Benchmark for Evaluating Large Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure