Integrating artificial intelligence with expert knowledge in global environmental assessments: opportunities, challenges and the way ahead

Veruska Muccione, Saeid Ashraf Vaghefi, Julia Bingler, Simon Allen, Mathias Kraus, Glen Gostlow, Tobias Wekhof, Chiara Colesanti-Senni, Dominik Stammbach, Jingwei Ni, Tobias Schimanski, Tingyu Yu, Qian Wang, Christian Huggel, Jürg Luterbacher, Robbert Biesbroek, Markus LeippoldRegional 2024 · DOI

Abstract

With new cycles of global environmental assessments (GEAs) recently starting, including GEO-7 and IPCC AR7, there is increasing need for artificial intelligence (AI) to support in synthesising the rapidly growing body of evidence for authors and users of these assessments. In this article, we explore recent advances in AI and connect them to the different stages of GEAs showing how some processes can be automatised and streamlined. The meticulous and labour-intensive nature of GEAs serves as both a valuable strength and a challenge to staying pertinent and current in today’s era of urgency and the pursuit of the latest knowledge. Utilising AI tools for reviewing and synthesizing scientific literature holds the evident promise of substantially lessening the workload for experts and expediting the assessment process. This, in turn, could lead to more frequent report releases and a smoother integration of the latest scientific advancements into actionable measures. However, successful outcomes can only be achieved if domain experts co-develop and oversee the deployment of such tools together with AI researchers. Otherwise, these tools run the risk of producing inaccurate, incomplete, or misleading information with significant consequences. We demonstrate this through a few examples that compare recently deployed large language models (LLMs) based tools in their performance in capturing nuanced concepts in the context of the reports of the Intergovernmental Panel on Climate Change (IPCC). We recommend establishing ethical committees and organising dedicated expert meetings to develop best practice guidelines, ensuring responsible and transparent integration of AI into GEAs.

1. Introduction

Global Environmental Assessments (GEAs), such as those conducted by the IPCC and IPBES, are critical for informing international policy. However, these assessments are currently facing an "information deluge." Between 2016 and 2021 alone, over 57,000 unique articles were published on climate change. For the IPCC’s 6th Assessment Report (AR6), authors had to manually evaluate over 66,000 publications. This labor-intensive process makes it increasingly difficult for reports to remain current in an era of rapid scientific advancement. This paper explores how artificial intelligence, specifically Large Language Models (LLMs), can streamline evidence synthesis while maintaining scientific rigor through expert oversight.

2. Methodology: The QA-Experts Workflow

The authors propose a "QA-Experts workflow" utilizing Retrieval-Augmented Generation (RAG). Unlike standard LLMs that rely on internal training data, RAG systems retrieve information from a curated, high-quality corpus.

The QA-Experts workflow integrates expert source selection, document parsing, semantic search, and expert verification to ensure accurate AI-generated answers.

The workflow consists of five key stages:

  • Curated Corpus: Experts select authoritative sources, such as IPCC reports and peer-reviewed literature.
  • Parsing & Indexing: Documents are segmented into "chunks" and stored in a vector database.
  • Semantic Search: User queries trigger the retrieval of the most relevant text segments.
  • Augmented Generation: An LLM (e.g., GPT-4) generates a response based on the retrieved segments, providing direct citations.
  • Expert Verification: Human experts review the output against original sources to prevent "hallucinations."

The study compared three configurations: a standard GPT-4 model, a "Hybrid" model (GPT-4 supplemented with IPCC data), and a "Standalone" model (restricted exclusively to IPCC data).

3. Key Insights: Nuance, Citations, and Accessibility

The research highlights several critical areas where AI can transform GEAs:

  • The Nuance Gap: LLMs can struggle with precise scientific terminology. For instance, models may conflate "acceleration" with "rate of change" if the source text lacks explicit mathematical definitions.
  • Traceability: The primary advantage of RAG is the provision of specific citations (e.g., page numbers). This allows researchers to immediately verify claims within massive reports.
  • Language Inclusivity: AI can help overcome English-language bias by translating non-English queries, processing them against English-centric databases, and translating the results back, broadening the evidence base.

4. Comparative Results and Stress Tests

The performance of these models was evaluated through various stress tests, focusing on their ability to handle complex climate queries.

Comparison of responses from GPT-4, a Hybrid model, and Standalone ChatClimate regarding the phase-out of fossil fuels.

Key findings from these tests include:

  • Instructional Depth: As shown in Figure 6, Standalone and Hybrid models provide significantly more detailed answers with specific IPCC citations compared to the generic responses of vanilla GPT-4.
  • Breadth of Knowledge: Models performed better when provided with the full suite of IPCC reports rather than just synthesis summaries, particularly for complex topics like "tipping points."
  • Tool Accuracy: Comparisons between specialized tools like ClimateQA and ChatClimate revealed differences in how they reflect the magnitude of historical emissions, emphasizing the need for continuous tool refinement.

5. Discussion and Future Directions

While AI offers transformative potential, several challenges remain. Models can still "hallucinate" information not present in the source text, and they currently struggle to interpret complex figures and tables. Furthermore, the environmental impact of training large models must be considered.

The authors conclude that AI should be viewed as a supplement to, not a replacement for, expert judgment. They recommend establishing ethical committees and best-practice guidelines to ensure the responsible integration of AI into global policy-making. Moving forward, these tools could enable "living reports" that are updated in real-time as new research emerges, ensuring that policy decisions are always based on the latest available evidence.

Summary updated

Abstract

Abstract With new cycles of global environmental assessments (GEAs) recently starting, including GEO-7 and IPCC AR7, there is increasing need for artificial intelligence (AI) to support in synthesising the rapidly growing body of evidence for authors and users of these assessments. In this article, we explore recent advances in AI and connect them to the different stages of GEAs showing how some processes can be automatised and streamlined. The meticulous and labour-intensive nature of GEAs serves as both a valuable strength and a challenge to staying pertinent and current in today’s era of urgency and the pursuit of the latest knowledge. Utilising AI tools for reviewing and synthesizing scientific literature holds the evident promise of substantially lessening the workload for experts and expediting the assessment process. This, in turn, could lead to more frequent report releases and a smoother integration of the latest scientific advancements into actionable measures. However, successful outcomes can only be achieved if domain experts co-develop and oversee the deployment of such tools together with AI researchers. Otherwise, these tools run the risk of producing inaccurate, incomplete, or misleading information with significant consequences. We demonstrate this through a few examples that compare recently deployed large language models (LLMs) based tools in their performance in capturing nuanced concepts in the context of the reports of the Intergovernmental Panel on Climate Change (IPCC). We recommend establishing ethical committees and organising dedicated expert meetings to develop best practice guidelines, ensuring responsible and transparent integration of AI into GEAs.
BibTeX
@inproceedings{muccione2024integrating,
  author = {Veruska Muccione and Saeid Ashraf Vaghefi and Julia Bingler and Simon Allen and Mathias Kraus and Glen Gostlow and Tobias Wekhof and Chiara Colesanti-Senni and Dominik Stammbach and Jingwei Ni and Tobias Schimanski and Tingyu Yu and Qian Wang and Christian Huggel and Jürg Luterbacher and Robbert Biesbroek and Markus Leippold},
  title = {{Integrating artificial intelligence with expert knowledge in global environmental assessments: opportunities, challenges and the way ahead}},
  booktitle = {Regional},
  year = {2024},
  doi = {10.1007/s10113-024-02283-8},
}
Figure