Assessing the Effectiveness of GPT-4o in Climate Change Evidence Synthesis and Systematic Assessments: Preliminary Insights

Elphin Joe, Sai Koneru, Christine KirchhoffACL 2024

TL;DR

This paper investigates how well GPT-4o can automate the complex task of synthesizing climate change adaptation evidence from scientific literature. While GPT-4o shows promise for simpler tasks like identifying geographic locations, its performance drops significantly for more nuanced assessments, highlighting the need for hybrid human-AI approaches.

What Problem Does This Paper Address?

The sheer volume of scientific literature on climate change adaptation makes it incredibly challenging for researchers and policymakers to keep up. Traditional methods of evidence synthesis, which rely on large teams of human experts manually reviewing and extracting information, are becoming unsustainable. This paper explores whether advanced Large Language Models (LLMs) like GPT-4o can help automate parts of this process, specifically focusing on extracting key information for systematic assessments, which are crucial for informing international treaties and national policies.

Methods

The researchers evaluated GPT-4o's ability to extract three types of information from scientific articles related to climate change adaptation:

  1. Geographic Location (Low Expertise): This involved identifying the specific country or region where an adaptation response was observed. This is akin to standard Named Entity Recognition (NER) tasks.
  2. Stakeholders (Intermediate Expertise): This required identifying which groups (e.g., government, civil society, individuals) were involved in the adaptation response, using a predefined taxonomy. This involves understanding the context and applying a classification scheme.
  3. Depth of Adaptation Response (High Expertise): This was the most challenging task, requiring an assessment of whether an adaptation response represented a minor adjustment to existing practices or a transformative change. This demands a deep understanding of adaptation literature and practical context.

GPT-4o was prompted to perform these tasks, and its outputs were compared against human annotations from the Global Adaptation Mapping Initiative (GAMI) dataset. The evaluation used precision, recall, and F1 scores.

Datasets

The study utilized a sample (n=586) from the Global Adaptation Mapping Initiative (GAMI) dataset. This dataset consists of information extracted from 1,682 peer-reviewed articles on climate change adaptation, with features labeled by climate change adaptation experts. For this study, the researchers focused on articles within the "food sector focus group" of the GAMI database.

Key Insights and Ideas

  • LLMs can automate parts of evidence synthesis, but expertise matters: GPT-4o demonstrates that LLMs can be useful tools for extracting information from scientific literature, but their effectiveness is highly dependent on the complexity of the task.
  • Low-expertise tasks are a strong suit: GPT-4o excels at straightforward information extraction, like identifying geographic locations, often providing more granular detail than human annotators.
  • Nuance and domain-specific understanding are challenges: For tasks requiring deeper comprehension, such as identifying stakeholders or assessing the transformative nature of adaptation responses, GPT-4o's performance falters. This suggests that current LLMs struggle with the subtle interpretations and contextual knowledge that human experts possess.
  • Human-AI collaboration is key: The findings strongly suggest that the future of evidence synthesis lies in hybrid approaches, where LLMs handle initial data processing and extraction, while human experts provide oversight, refinement, and handle the more complex analytical tasks.

Key Results

  • Geographic Location: GPT-4o achieved high performance with a precision of 0.88 and recall of 0.9. It often provided more specific country-level details than human annotators.
  • Stakeholders: Performance dropped significantly, with a micro F1 score of 0.54. GPT-4o sometimes misidentified or extracted irrelevant stakeholders mentioned in other parts of the text.
  • Depth of Adaptation Response: This was the most challenging task, with an accuracy of 22.7% and a micro-averaged F1 score of 0.22. GPT-4o tended to be overly optimistic, potentially overestimating the impact of adaptation responses, indicating a struggle with nuanced qualitative assessment.

Limitations and Future Work

The authors acknowledge several limitations:

  • Scope: The study was limited to the food sector and did not explore other climate change adaptation domains.
  • Prompting Techniques: Complex prompting strategies like "Chain of Verification" were not explored, which might improve accuracy.
  • Model Specificity: The evaluation was solely on GPT-4o; comparisons with other LLMs are needed.

Future work should focus on:

  • Developing methods to integrate domain knowledge and human feedback to improve GPT-4o's performance on medium and high-expertise tasks.
  • Exploring advanced prompting techniques to enhance reliability.
  • Conducting comparative studies with multiple LLMs.
  • Developing human-in-the-loop systems that leverage LLMs' strengths while ensuring the integrity and effectiveness of the expert-driven process.

Summary updated

BibTeX
@inproceedings{joe2024assessing,
  author = {Elphin Joe and Sai Koneru and Christine Kirchhoff},
  title = {{Assessing the Effectiveness of GPT-4o in Climate Change Evidence Synthesis and Systematic Assessments: Preliminary Insights}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure