Towards unearthing neglected climate innovations from scientific literature using Large Language Models

César Quilodrán-Casas, Christopher Waite, Nicole Alhadeff, Diyona Dsouza, Cathal Hughes, Larissa Kunstel-Tabet, Alyssa Gilbert

TL;DR

This paper explores using Large Language Models (LLMs) to automatically identify promising but overlooked climate innovations within scientific literature. By comparing LLM evaluations to human assessments, they demonstrate LLMs can significantly accelerate the discovery of impactful climate solutions, paving the way for faster climate action.

What Problem Does This Paper Address?

The urgent need to address climate change requires rapid identification and deployment of innovative solutions. However, the sheer volume and interdisciplinary nature of scientific literature make it challenging to discover these innovations. Many potentially impactful solutions may be buried within existing research, underutilized due to traditional knowledge synthesis methods' limitations. This is particularly true for countries like the UK, which invest heavily in climate research but may not be fully leveraging their existing scientific output. The paper aims to bridge this gap by developing a systematic, AI-driven approach to unearth these neglected climate innovations.

Methods

The researchers developed a workflow to identify and evaluate climate innovations using LLMs. This involved:

  • Data Collection: Sourcing academic papers from the OpenAlex database, focusing on UK institutions and excluding health/social sciences to concentrate on technical research.
  • LLM Evaluation: Employing GPT-4o to analyze paper abstracts across seven dimensions: mitigation potential, technology stage, readiness, market need, tech enabling potential, eco-focus, and neglectedness.
  • LLM Scenarios: Testing the LLM's performance under three conditions:
  • No-shot: Abstract only.
  • Context: Abstract plus human-provided context explaining each evaluation dimension.
  • Few-Shot Learning: Abstract plus examples of successful climate tech spin-outs.
  • Human Evaluation: A survey with six participants who independently evaluated the same abstracts and dimensions.
  • Comparison and Analysis: Comparing LLM and human responses using Cohen's Kappa for agreement and Pearson correlation for relationships between questions.
  • Ranking Algorithm: Developing a preliminary algorithm that first filters abstracts based on mitigation potential (Q1) and then uses weighted scores from other dimensions (derived from logistic regression on positive controls) to rank potential innovations.
  • Scalar Scoring: Investigating the use of a 1-10 scalar scoring system instead of binary (yes/no) to improve ranking granularity.

Datasets

  • OpenAlex: A comprehensive, open, community-curated database of scholarly publications used to extract the initial dataset of academic papers.
  • Curated Test Set: A subset of 100 abstracts randomly selected from the OpenAlex data, including 5 "positive control" abstracts from UK-based institutions that led to spin-out climate-tech companies.
  • Larger Validation Dataset: A set of 1000 abstracts, including an additional 10 positive control abstracts, used to validate the optimized LLM search tool.

Key Insights and Ideas

  • LLMs as Accelerators: LLMs can significantly augment human expertise in identifying climate innovations, offering greater speed, throughput, and consistency.
  • Context Matters: Providing LLMs with contextual information (like the purpose of each evaluation question) improves their alignment with human judgments.
  • Few-Shot Learning's Nuance: While few-shot learning with examples of successful spin-outs was tested, its impact on improving LLM performance was less pronounced than providing direct contextual explanations.
  • LLM Reasoning Patterns: LLMs tend to exhibit more deterministic reasoning than humans. They also appear to group questions into distinct clusters: those related to technical/market readiness versus those related to early-stage, potentially neglected research.
  • Scalar Scoring for Granularity: Moving from binary to scalar scoring (1-10) for LLM evaluations significantly improves the resolution of ranking potential innovations, making it easier to differentiate between high-scoring abstracts.
  • Neglectedness is Tricky: LLMs, especially with context, sometimes scored commercially successful technologies lower on "neglectedness," which is counter-intuitive but reflects the reality that successful innovations are, by definition, no longer neglected.

Key Results

  • Positive Controls Identified: All three LLM scenarios successfully identified the 5 positive control abstracts, arguably more convincingly than human participants.
  • Agreement with Humans: The LLM with context (binary) showed the most similarity to human scoring, particularly for technology readiness and neglectedness.
  • Cohen's Kappa: Moderate agreement was observed for mitigation and eco-focus questions, with the context scenario yielding the highest Kappa scores. Agreement was lower for technological enabling and neglectedness.
  • Ranking Improvement: The scalar scoring system, combined with weighted scores, significantly improved the ranking resolution, reducing the number of indistinguishable abstracts from 12 to 2 out of 24 passed abstracts.
  • Validation Success: In a larger validation set of 1000 abstracts, the optimized LLM approach identified 8 out of 10 positive control abstracts, ranked within the top 95.

Limitations and Future Work

  • Limited Positive Controls: The small number of positive control abstracts (5 for initial weighting, 10 for validation) limits the robustness of the weighting algorithm. Training on a larger set of successful innovations could improve predictive quality.
  • LLM Determinism: The highly deterministic nature of LLM responses, while consistent, can lead to a lack of nuanced differentiation in rankings, especially with binary scoring.
  • Future Work:
    • Experimenting with different LLMs (e.g., LLaMA) and domain-specific models.
  • Incorporating Retrieval-Augmented Generation (RAG) for more extensive context.
  • Developing multi-agent approaches for more robust consensus in abstract assessment.
  • Building a relational layer to link papers with authors, grants, and social media to identify entrepreneurial potential.
  • Applying the workflow to the entire UK corpus and potentially other regions.

Summary updated

Abstract

Climate change poses an urgent global threat, needing the rapid identification and deployment of innovative solutions. We hypothesise that many of these solutions already exist within scientific literature but remain underutilised. To address this gap, this study employs a curated dataset sourced from OpenAlex, a comprehensive repository of scientific papers. Utilising Large Language Models (LLMs), such as GPT4-o from OpenAI, we evaluate title-abstract pairs from scientific papers on seven dimensions, covering climate change mitigation potential, stage of technological development, and readiness for deployment. The outputs of the language models are then compared with human evaluations to assess their effectiveness in identifying promising yet overlooked climate innovations. Our findings suggest that these LLM-based models can effectively augment human expertise, uncovering climate solutions that are potentially impactful but with far greater speed, throughput and consistency. Here, we focused on UK-based solutions, but the workflow is region-agnostic. This work contributes to the discovery of neglected innovations in scientific literature and demonstrates the potential of AI in enhancing climate action strategies.
BibTeX
@inproceedings{quilodrancasas2024towards,
  author = {César Quilodrán-Casas and Christopher Waite and Nicole Alhadeff and Diyona Dsouza and Cathal Hughes and Larissa Kunstel-Tabet and Alyssa Gilbert},
  title = {{Towards unearthing neglected climate innovations from scientific literature using Large Language Models}},
  booktitle = {LXAI @ NeurIPS},
  year = {2024},
  url = {https://openreview.net/forum?id=j4rSjbDPrB},
}
Figure