Example-Based Concept Analysis Framework for Deep Weather Forecast Models

Deep Neural Networks (DNNs) are revolutionizing weather forecasting, often outperforming traditional numerical models in speed and accuracy. However, for a meteorologist responsible for public safety, a "black box" prediction isn't enough—they need to understand the underlying physical mechanism. This framework, developed by researchers from KAIST and the Korea Meteorological Agency (KMA), bridges this gap by turning abstract vector representations into human-understandable meteorological concepts.

Addressing the "Black Box" Problem

While DNNs excel at pattern recognition, their internal decision-making process is opaque. In high-stakes fields like meteorology, trust is paramount. Existing XAI methods, such as heatmaps, are often "AI-centric"—useful for developers but difficult for domain experts to translate into actionable scientific insights. Forecasters need to know if a model's prediction of heavy rain is based on a valid meteorological mechanism (e.g., a convectional cell) or a data artifact. This framework provides user-centric tools that map high-dimensional model features to the specific vocabulary and mental models used by professional meteorologists.

Methodology

The framework consists of three core technological pillars:

1. Probabilistic Concept Probers: Binary SVM classifiers detect specific meteorological concepts within the model's bottleneck layer. These probers provide a probability score, helping resolve ambiguities in complex weather patterns.
2. Principal Neuron Component-based Neighbor Search (PC-NSE): To show forecasters similar historical examples, the system identifies "Principal Neuron Components"—the specific neurons most responsible for representing a concept. This reduced the search dimensionality from ~1.65 million to just 300.
3. Modified Importance Scores: The standard TCAV (Testing with Concept Activation Vectors) method was adapted for segmentation models. Wrapper functions aggregate how much a specific concept influences the final rainfall intensity prediction across the entire map.

Expand

Source, p. 6

Datasets

The study utilized two primary data sources:

• Radar Hybrid Surface Rainfall (HSR): 10-minute interval composited radar data from the Korean Peninsula (2018–2021).
• Human-Annotated Concept Labels: A new dataset compiled from KMA and NIMS reports, including 63 to 82 distinct meteorological concepts (e.g., "sea breeze convergence," "typhoon," "changma front") annotated by experts.

Key Insights

• DNNs as "Accidental" Meteorologists: DNNs naturally encode semantically meaningful meteorological attributes in their internal layers without explicit physics training.
• Non-Orthogonal Features: The "Principal Neuron" approach, which doesn't force orthogonality, is more effective for capturing physically interconnected weather systems than standard PCA.
• XAI for Debugging: The framework revealed model confusion between "bright band" radar errors and actual convection, allowing engineers to identify data quality issues.
• User-Centricity: The dashboard mirrors human workflow by showing "Top 3 Nearest Neighbors" and "Top 5 Concepts," providing explanations via historical "analog" cases.

Performance and Results

The calibrated SVM ensemble achieved a Macro F1 score of 0.77. Crucially, the PC-NSE search engine significantly improved operational efficiency.

Expand

Source, p. 8

As shown in the results, the PC-NSE method reduced query time from 6.8 seconds to 1.5 seconds while maintaining competitive precision compared to the full latent space $Z$. Importance scores also indicated that the model is most sensitive to high-impact events like typhoons and low-level jets.

Limitations and Future Work

The current dataset is specific to the Korean Peninsula, and expanding it would require further expert annotations. Future work aims to incorporate more variables (e.g., thermal instability) to move toward causal explanations and extend the framework to generative weather models like Diffusion models.

@misc{kim2025example,
  author = {Soyeon Kim and Junho Choi and Subeen Lee and Jaesik Choi},
  title = {{Example-Based Concept Analysis Framework for Deep Weather Forecast Models}},
  year = {2025},
  eprint = {2504.00831},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.00831},
}