Vision-Language Models Meet Meteorology: Developing Models for Extreme Weather Events Detection with Heatmaps

Jian Chen, Peilin Zhou, Yining Hua, Dading Chong, Meng Cao, Yaowei Li, Zixuan Yuan, Bing Zhu, Junwei LiangarXiv 2024 · arXiv · OpenReview

Abstract

Real-time detection and prediction of extreme weather protect human lives and infrastructure. Traditional methods rely on numerical threshold setting and manual interpretation of weather heatmaps with Geographic Information Systems (GIS), which can be slow and error-prone. Our research redefines Extreme Weather Events Detection (EWED) by framing it as a Visual Question Answering (VQA) problem, thereby introducing a more precise and automated solution. Leveraging Vision-Language Models (VLM) to simultaneously process visual and textual data, we offer an effective aid to enhance the analysis process of weather heatmaps. Our initial assessment of general-purpose VLMs (e.g., GPT-4-Vision) on EWED revealed poor performance, characterized by low accuracy and frequent hallucinations due to inadequate color differentiation and insufficient meteorological knowledge. To address these challenges, we introduce ClimateIQA, the first meteorological VQA dataset, which includes 8,760 wind gust heatmaps and 254,040 question-answer pairs covering four question types, both generated from the latest climate reanalysis data. We also propose Sparse Position and Outline Tracking (SPOT), an innovative technique that leverages OpenCV and K-Means clustering to capture and depict color contours in heatmaps, providing ClimateIQA with more accurate color spatial location information. Finally, we present Climate-Zoo, the first meteorological VLM collection, which adapts VLMs to meteorological applications using the ClimateIQA dataset. Experiment results demonstrate that models from Climate-Zoo substantially outperform state-of-the-art general VLMs, achieving an accuracy increase from 0% to over 90% in EWED verification.

The Challenge of Meteorological VQA

While state-of-the-art Vision-Language Models (VLMs) like GPT-4o or LLaVA excel at general image description, they struggle significantly with "AI for Science" tasks involving meteorological heatmaps. These visualizations present unique challenges: irregular contours, unstructured patterns, and color gradients where subtle hue shifts represent significant physical changes in wind speed or precipitation.

The authors identified three critical failures in current models:

  1. Color Misinterpretation: Models often confuse similar colors (e.g., mistaking yellow for red), leading to incorrect severity assessments.
  2. Hallucinations: Models invent weather details that aren't in the data.
  3. Incomplete Spatial Reasoning: Models struggle to "see" the boundaries of a storm or provide precise geographic coordinates for an anomaly.

Methodology: SPOT and Climate-Zoo

To address these challenges, the paper introduces two major technical contributions:

SPOT (Sparse Position and Outline Tracking)

SPOT is a novel algorithm that bridges the gap between raw pixels and structured geographic data. It uses color segmentation (via OpenCV) to isolate regions of interest and applies K-Means clustering to find representative "anchor points" within those regions. This converts a chaotic "blob" of color into a set of precise coordinates that a language model can process, providing a spatial "skeleton" that drastically reduces localization errors.

Climate-Zoo

The researchers developed Climate-Zoo, a suite of models based on leading VLM architectures such as Qwen-VL, LLaVA 1.6, and Yi-VL. These models were adapted using Supervised Fine-Tuning (SFT) and Low-Rank Adaptation (LoRA) on the ClimateIQA dataset. The training focused on four task types—Verification, Enumeration, Geo-Indexing, and Description—teaching the models a domain-specific "Chain-of-Thought" for weather analysis.

The ClimateIQA Dataset

The core of this work is ClimateIQA, a domain-specific VQA dataset designed to benchmark and train models for meteorological tasks:

  • Scale: 26,280 high-resolution heatmaps and 762,120 instruction-answer pairs.
  • Source: Built on the gold-standard ERA5 reanalysis dataset from 2023.
  • Content: Covers wind gusts (using the Beaufort Scale), total precipitation, and combined indices like Wind Chill and Heat Index.
  • Grounding: Integrates geographic metadata from IHO Sea Areas and World Bank administrative boundaries to ensure models learn real-world locations.

Key Results and Insights

Climate-Zoo models achieved approximately 91% accuracy in pinpointing anomalies, whereas baseline models like GPT-4o often failed to identify more than 15-22% of critical regions.

Comparison of general-purpose VLMs (LLaVA, GPT-4o) and Climate-Zoo on extreme weather detection tasks.

A notable finding was the performance of the Yi-VL-6B model, which reached peak performance with only 10,000 samples. Researchers hypothesize that its pre-training on a massive "encyclopedic" corpus gave it a superior baseline knowledge of geography and meteorology. Additionally, the study found that "Verification" tasks are foundational; removing them from training crippled the model's ability to generate detailed "Descriptions."

Limitations and Future Work

Despite the progress, several hurdles remain. Models still occasionally struggle with fine-grained color identification in complex global maps. The authors suggest splitting heatmaps into sub-images (tiling) as a potential fix to improve focus and resolution.

Experimental strategies to improve color identification and spatial reasoning, including image splitting and grid overlays.

Other areas for future improvement include:

  • Projection Distortions: Mercator-style projections introduce distortions at the poles. Future work aims to train models directly on raw numerical/grid data.
  • Empirical Parameters: The SPOT algorithm relies on manually chosen parameters for clustering, which may require further automation.
  • Expanding Scope: The team plans to expand the dataset to include phenomena like droughts and typhoons and integrate multi-modal sources like satellite imagery and ground sensors.

Summary updated

Abstract

Real-time detection and prediction of extreme weather protect human lives and infrastructure. Traditional methods rely on numerical threshold setting and manual interpretation of weather heatmaps with Geographic Information Systems (GIS), which can be slow and error-prone. Our research redefines Extreme Weather Events Detection (EWED) by framing it as a Visual Question Answering (VQA) problem, thereby introducing a more precise and automated solution. Leveraging Vision-Language Models (VLM) to simultaneously process visual and textual data, we offer an effective aid to enhance the analysis process of weather heatmaps. Our initial assessment of general-purpose VLMs (e.g., GPT-4-Vision) on EWED revealed poor performance, characterized by low accuracy and frequent hallucinations due to inadequate color differentiation and insufficient meteorological knowledge. To address these challenges, we introduce ClimateIQA, the first meteorological VQA dataset, which includes 8,760 wind gust heatmaps and 254,040 question-answer pairs covering four question types, both generated from the latest climate reanalysis data. We also propose Sparse Position and Outline Tracking (SPOT), an innovative technique that leverages OpenCV and K-Means clustering to capture and depict color contours in heatmaps, providing ClimateIQA with more accurate color spatial location information. Finally, we present Climate-Zoo, the first meteorological VLM collection, which adapts VLMs to meteorological applications using the ClimateIQA dataset. Experiment results demonstrate that models from Climate-Zoo substantially outperform state-of-the-art general VLMs, achieving an accuracy increase from 0% to over 90% in EWED verification. The datasets and models in this study are publicly available for future climate science research: https://github.com/AlexJJJChen/Climate-Zoo.
BibTeX
@misc{chen2024vision,
  author = {Jian Chen and Peilin Zhou and Yining Hua and Dading Chong and Meng Cao and Yaowei Li and Zixuan Yuan and Bing Zhu and Junwei Liang},
  title = {{Vision-Language Models Meet Meteorology: Developing Models for Extreme Weather Events Detection with Heatmaps}},
  year = {2024},
  eprint = {2406.09838},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.09838},
}
Figure