Satellite-Text-Prompted Large Language Model for Photovoltaic Power Forecasting

Pengfei Jia, Jianghong Ma, Baoquan Zhang, Kenghong Lin, Xinyu Zhang, Chuyao Luo, Xutao Li, Yunming Ye

TL;DR

PV-LLM is a novel forecasting framework that treats solar power prediction as a multimodal "reasoning" task. By integrating satellite imagery and weather data with textual metadata (like plant coordinates and time) into a pretrained Large Language Model (GPT-2), the system achieves state-of-the-art accuracy, particularly in "hard" scenarios where weather is highly unpredictable.

What Problem Does This Paper Address?

Photovoltaic (PV) power forecasting is notoriously difficult because solar output is highly intermittent, fluctuating wildly based on cloud cover and local weather. While current state-of-the-art models use Numerical Weather Prediction (NWP) and satellite imagery, they often struggle with two things:

  1. The Modality Gap: It is difficult to fuse raw numbers (weather) with 2D grids (satellite images) in a way that captures the "big picture."
  2. Contextual Knowledge: Human forecasters use high-level context—knowing exactly where a plant is located or what time of day it is—to refine their guesses. Standard neural networks often treat these as just more numbers, missing the semantic "meaning" (e.g., "it's noon in a coastal region") that could improve peak power predictions.

Methods

The authors propose PV-LLM, which uses a pretrained LLM as a "reasoning engine" to fuse three distinct types of data:

  • Text Prompt Construction (TPC): This module converts metadata into natural language. Instead of feeding the model raw coordinates, it generates a structured prompt: "Target coordinates are [Lat, Lon]... Starting at [Time]..." This activates the LLM’s internal knowledge about geographic and temporal patterns.
  • Modality-Specific Encoding (MSE):
    • Temporal-Semantic Encoder: Processes PV and weather time-series. It uses a "Semantic Adapter" to map these numerical sequences into the same "word embedding space" the LLM uses, essentially translating numbers into the LLM's vocabulary.
  • Satellite Image Sequence Encoder: Uses Rotary Positional Encoding (RoPE) to help the model understand the spatial relationship between different parts of a satellite image. It also includes a "Context Compressor" to ensure the massive amount of visual data doesn't overwhelm the LLM's limited context window.
  • Adaptive Prompt Tuning (APT): Rather than just shoving everything into the first layer, the authors inject satellite information into intermediate layers of the LLM. They use an adaptive gating mechanism initialized to zero, which allows the model to gradually learn how much weight to give the satellite data without "corrupting" the LLM's pretrained knowledge during early training.

Datasets

The model was evaluated on the MMSP(S) dataset, which includes:

  • PV Power Data: Ground-truth measurements from 10 plants in Shandong Province, China.
  • Satellite Imagery: High-resolution cloud dynamics from the Himawari-8/9 satellites.
  • NWP Data: Meteorological predictions from the European Centre for Medium-Range Weather Forecasts (ECMWF). The data spans 1.5 years (Jan 2021 – June 2022) with a one-hour temporal resolution.

Key Insights and Ideas

  • Language as a Universal Glue: The most striking insight is that describing a power plant's location and time in words helps the model more than just providing them as numerical features. This suggests LLMs can leverage their "common sense" about geography and time to better interpret weather data.
  • The "Semantic Adapter" is Essential: The authors found that simply feeding numbers to an LLM doesn't work. You must map those numbers into "word prototypes." This bridges the gap between the continuous world of physics and the discrete world of language.
  • Stability through Zero-Initialization: By starting the adaptive gates at zero, the researchers ensured the LLM didn't "forget" its language capabilities before it learned how to handle the new satellite inputs. This "slow-burn" integration was key to their performance gains.

Key Results

PV-LLM outperformed 15 baseline methods, including specialized time-series models (like PatchTST) and other multimodal fusion networks (like FusionSF).

  • Overall Performance: Achieved the lowest Mean Absolute Error (MAE) of 0.03962 across the entire dataset.
  • Robustness in "Hard" Scenarios: In cases where power output deviated significantly from historical patterns (e.g., sudden storms or unusual cloud cover), PV-LLM showed the most significant improvement over traditional methods, proving that the multimodal approach provides a much-needed "safety net" for unpredictable weather.

Limitations and Future Work

  • Computational Overhead: Using an LLM (even a relatively small one like GPT-2) is computationally more expensive than using a simple MLP or Transformer. Real-time deployment would require significant hardware resources.
  • Geographic Specificity: The study focused on a single province in China. It remains to be seen how well the "textual reasoning" transfers to vastly different climates (e.g., tropical vs. arid regions) without extensive retraining.
  • Future Directions: The authors suggest exploring even larger LLMs and investigating if the model can generate "natural language explanations" for its forecasts, which would be a massive boon for power grid operators.

Summary updated

Abstract

Photovoltaic (PV) power forecasting is critical for the operation of solar power plants and the coordination of energy within power grids. This work aims to predict future PV power time series by leveraging multimodal data. While recent studies have incorporated numerical modalities such as satellite image sequences and numerical weather prediction (NWP) time series, they often overlook textual modalities—such as the spatio-temporal context of PV plants—and the potential of pretrained large language models (LLMs). In this paper, we build upon existing numerical inputs and further explore the use of spatio-temporal text prompts, generated based on plant coordinates and forecast start time, to enhance the forecasting process. We propose PV-LLM, a satellite-text-prompted framework that integrates a pretrained LLM to improve PV power forecasting. The framework consists of three key components: Text Prompt Construction, Modality-Specific Encoding, and Adaptive Prompt Tuning. First, the Text Prompt Construction module generates spatio-temporal prompts that offer high-level semantic guidance. Next, the Modality-Specific Encoding module encodes each modality according to its unique characteristics, capturing modality-specific patterns while managing varying context lengths. Finally, the Adaptive Prompt Tuning module fine-tunes the LLM to integrate multimodal embeddings, while an adaptive gating mechanism retains its pretrained knowledge. We validate the effectiveness of the proposed framework on a real-world dataset containing multiple PV plants. Experimental results demonstrate that our approach outperforms existing state-of-the-art methods.
BibTeX
@inproceedings{jia2026satellite,
  author = {Pengfei Jia and Jianghong Ma and Baoquan Zhang and Kenghong Lin and Xinyu Zhang and Chuyao Luo and Xutao Li and Yunming Ye},
  title = {{Satellite-Text-Prompted Large Language Model for Photovoltaic Power Forecasting}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i45.41207},
}

Loading related content...

Figure