Cross-Domain Generalization of Multimodal LLMs for Global Photovoltaic Assessment

Muhao Guo, Yang WengarXiv 2025 · DOI · arXiv

Introduction

The rapid expansion of distributed photovoltaic (PV) systems, such as residential solar panels, is critical for the green energy transition but poses a significant challenge for grid operators. Many installations remain undocumented, making grid management difficult. While satellite imagery offers a scalable solution for detection, traditional computer vision (CV) models like CNNs or U-Nets are often brittle, failing to generalize across different architectural styles and climates. This paper introduces the PV Assessment with LLMs (PVAL) framework, which leverages the semantic reasoning of multimodal Large Language Models (LLMs) to achieve robust, global-scale solar panel detection and quantification.

Methods

The PVAL framework treats solar detection as a multimodal inference problem rather than a simple image segmentation task. The process is divided into four key stages:

  1. Data Engineering: High-resolution satellite images are sliced into 4×4 grids. These tiles are manually annotated for presence, spatial localization, and quantity.
  2. Prompt Engineering: A structured system prompt enforces task decomposition (analysis, localization, then quantification) and requires the model to return results in a strict JSON schema.
  3. Supervised Fine-Tuning (SFT): GPT-4o is fine-tuned on a specific subset of North American data to map visual features to the required assessment labels.
  4. Few-Shot Re-anchoring: In-context examples are used to maintain consistency and "anchor" the model's reasoning when applied to new geographic domains.
Overview of the PVAL framework, including data engineering, prompt engineering, and the global evaluation regions.

Datasets and Evaluation

The model was trained on 2,000 annotated tiles from Santa Ana, CA. To test its cross-domain generalization, the researchers compiled an evaluation set of 480 tiles from each of six geographically diverse cities: Sydney (Australia), Cape Town (South Africa), Kuwait City (Kuwait), Oxford (UK), São Paulo (Brazil), and Shanghai (China). This variety ensures the model is tested against different roof geometries, lighting conditions, and urban densities.

Key Insights and Results

PVAL outperforms traditional models by utilizing "world knowledge" to distinguish solar panels from visually similar objects like skylights or shadows. Unlike standard pipelines that require separate models for detection and counting, PVAL performs these tasks natively through structured text generation.

The spatial localization grid and examples of multi-task outputs (detection, localization, and counting) generated by the model.

Performance Metrics

  • High Accuracy: In the training region, PVAL achieved a Precision of 0.961 and an F1-Score of 0.950, surpassing ResNet-152 and ViT-Base.
  • Global Robustness: The model showed significantly lower performance degradation ($\Delta F1$) when moving to unseen international cities. For instance, it saw only a -0.03 change in F1 for Oxford, whereas traditional models experienced sharp declines.
  • Functional Reasoning: The framework achieved 87.4% accuracy in localization and 80.6% in quantity estimation on the fine-tuning dataset.

Conclusion

The PVAL framework demonstrates that multimodal LLMs can overcome the generalization gap that plagues traditional computer vision in remote sensing. While challenges remain regarding very small or shaded panels, the ability to reason semantically about satellite imagery opens new doors for global infrastructure assessment and power system management. Future work will focus on integrating these insights into real-time decision-making tools for grid stability.

Summary updated

Abstract

The rapid expansion of distributed photovoltaic (PV) systems poses challenges for power grid management, as many installations remain undocumented. While satellite imagery provides global coverage, traditional computer vision (CV) models such as CNNs and U-Nets require extensive labeled data and fail to generalize across regions. This study investigates the cross-domain generalization of a multimodal large language model (LLM) for global PV assessment. By leveraging structured prompts and fine-tuning, the model integrates detection, localization, and quantification within a unified schema. Cross-regional evaluation using the $Δ$F1 metric demonstrates that the proposed model achieves the smallest performance degradation across unseen regions, outperforming conventional CV and transformer baselines. These results highlight the robustness of multimodal LLMs under domain shift and their potential for scalable, transferable, and interpretable global PV mapping.
BibTeX
@misc{guo2025cross,
  author = {Muhao Guo and Yang Weng},
  title = {{Cross-Domain Generalization of Multimodal LLMs for Global Photovoltaic Assessment}},
  year = {2025},
  eprint = {2511.19537},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.19537},
}
Figure