Evaluating Sentiment and Factuality of Offshore Wind Technological Trends Using Large Language Models

TL;DR

This paper introduces EnergyEval, a novel multi-agent framework using Large Language Models (LLMs) to analyze sentiment and factuality in energy literature. By employing a "debate" among specialized LLM agents, EnergyEval accurately forecasts offshore wind technology trends, aligning with real-world data and offering a more efficient and robust approach than single-LLM methods.

What Problem Does This Paper Address?

The global push for net-zero emissions requires rapid adoption of new energy technologies. To guide policy, investment, and deployment, accurately forecasting these technological trends is crucial. However, existing methods struggle to leverage the vast, unlabelled energy literature. This paper addresses this gap by proposing a novel way to extract actionable insights from this literature using LLMs, specifically focusing on sentiment and factuality to predict future developments in the offshore wind sector.

Methods

The core innovation is the EnergyEval framework, a multi-agent system. It utilizes heterogeneous, energy-specialized LLM agents that engage in a multi-agent debate (MAD). This approach leverages different LLMs with distinct roles (e.g., Policy Agent, Technology Agent) to critically evaluate and debate the sentiment and factuality of text. The framework incorporates one-shot chain-of-thought (CoT) prompting and a least-to-most strategy to enhance reasoning accuracy and mitigate hallucinations. The system first identifies topics and subtopics within sentences, then analyzes sentiment (polarity) and subjectivity.

Datasets

The study uses a dataset of approximately 5033 sentences from UK offshore wind literature collected between 2010 and 2022. This includes government and industry reports, academic journal articles, and policy briefings. A subset was used to create a ground truth dataset for evaluating LLM performance, involving human annotators and LLM-generated responses.

Key Insights and Ideas

• LLMs as a powerful tool for energy literature analysis: The study demonstrates that LLMs can effectively extract nuanced sentiment and factuality from technical energy literature, a task previously challenging for traditional methods.
• Multi-agent debate (MAD) enhances accuracy and reduces costs: By having specialized LLM agents debate, the framework achieves competitive accuracy while significantly reducing computational costs compared to single, powerful LLMs.
• Heterogeneous roles are key to robust analysis: Assigning specific energy-domain roles to LLM agents (e.g., Policy, Technology) improves the accuracy and relevance of sentiment analysis, especially for specialized technical content.
• Sentiment analysis can predict real-world trends: The sentiment extracted by EnergyEval closely aligns with observed trends in offshore wind capacity and cost reduction, suggesting its predictive power.
• Objective sentiment is more reliable: Combining objective sentiment with positive sentiment provides a more stable and accurate reflection of real-world trends than relying solely on positive sentiment.

Key Results

• EnergyEval's sentiment predictions for offshore wind technology trends (cost, capacity, turbine size) showed strong alignment with actual historical data from 2008-2020.
• The multi-agent debate approach, specifically using Mixtral as the generator and GPT-3.5 as the debater, achieved high accuracy in polarity and subjectivity tasks at a significantly lower cost than self-consistency methods.
• The positive-objective sentiment measurement proved more accurate and less prone to fluctuations than solely positive sentiment when compared to real-world data.
• Key drivers identified for offshore wind development were cost, capacity, and turbine size, with their influence on policy, technology, market, and supply chain sectors also analyzed.

Limitations and Future Work

• Data Representation: The study acknowledges that a lack of diverse literature representation can lead to unexpected fluctuations in forecasted trends. Future work should focus on gathering a wider variety of sources and ensuring sufficient data per year.
• Ground Truth Construction: While LLMs were used to aid in ground truth creation for efficiency, a hybrid approach with human oversight was employed. Future studies might explore different annotation strategies.
• Sentence-Level Limitations: The current sentence-level analysis doesn't fully capture document-level sentiment. Future research could incorporate an additional LLM layer for document-level sentiment extraction.
• Debate Rounds: The study did not explore the impact of increasing the number of debate rounds, which could further enhance accuracy.
• Smaller LLMs: Investigating whether smaller, energy-efficient LLMs can replace larger ones within the MAD framework is a promising avenue to reduce costs and environmental impact.
• Broader Applications: The framework's modular design suggests potential for extension to other net-zero technologies and multi-modal LLMs (e.g., analyzing podcasts, images).

@inproceedings{freed2025evaluating,
  author = {Holly Freed and Konstantina Vogiatzaki and Stephen Roberts},
  title = {{Evaluating Sentiment and Factuality of Offshore Wind Technological Trends Using Large Language Models}},
  booktitle = {Energies},
  year = {2025},
  doi = {10.3390/en18215816},
}