TL;DR
This paper tackles the challenge of automatically identifying Sustainable Development Goal (SDG) targets within environmental impact assessment (EIA) reports. They propose and evaluate a Retrieval-Augmented Generation (RAG) framework using Large Language Models (LLMs) to detect which SDG targets are addressed by specific text passages and to find textual evidence for a given SDG target. This work is crucial for streamlining the laborious process of environmental reporting and ensuring compliance with global sustainability goals.
What Problem Does This Paper Address?
Environmental assessment practitioners are increasingly required to demonstrate how their reports align with the United Nations' Sustainable Development Goals (SDGs), particularly specific SDG targets. Manually sifting through lengthy and complex environmental reports to find relevant information for each SDG target is a time-consuming and error-prone task. This paper addresses this gap by developing methods to automate two key sub-tasks:
- SDG Target Evidence Identification (EI): Determining if a given text passage in an EIA report provides evidence for a specific SDG target.
- SDG Target Detection (TD): Identifying which SDG targets are addressed within a given text passage from an EIA report.
The goal is to leverage the power of LLMs within a RAG framework to make this process more efficient and accurate.
Methods
The core of the paper's methodology is the Retrieval-Augmented Generation (RAG) framework. This approach combines information retrieval with text generation by LLMs. For each of the two tasks (EI and TD), the RAG pipeline consists of three main components:
- Retrieval: This component takes a query (either an SDG target for EI or a text passage for TD) and retrieves a ranked list of relevant passages or targets from an indexed collection. The paper explores two retrieval methods:
- Lexical Retrieval: Traditional keyword-based matching.
- Dense Retrieval: Using learned embeddings for semantic matching.
- Augmentation: The retrieved passages/targets are then used to augment the input to the LLM. This involves carefully crafting a prompt that includes the original query, the retrieved items, and specific instructions for the LLM. The paper investigates different augmentation strategies:
- Prompt Type: "SDG-explicit" (clearly mentioning SDG targets) vs. "SDG-implicit" (masking SDG targets as general "environmental policies").
- Number of Examples: Using one or two examples in the prompt for few-shot learning.
- Order of Passages: Presenting retrieved items in their ranked order or in a random order.
- Generation: A Large Language Model (LLM) then generates the answer based on the augmented prompt. The paper experiments with several LLMs:
- Open LLMs: Llama2-13b, Llama2-13b-ch, ClimateGPT-13b.
- Closed LLM: ChatGPT (gpt-3.5-turbo-0125).
Datasets
The authors introduce two new test collections for evaluating their methods:
- EI Test Collection: Consists of 30 selected SDG targets, each annotated with binary relevance judgments for 6 passages (the top 3 retrieved contexts from two different retrieval methods).
- TD Test Collection: Consists of 50 EIA report excerpts, each annotated with binary relevance judgments for 10 "passages" (which are actually the top 5 retrieved SDG targets for each of the two retrieval methods).
These datasets were derived from 33 environmental reports from Estonia and neighboring European countries. The full list of links to the PDF documents and the processed passage collection are made publicly available.
Key Insights and Ideas
- RAG is a Promising Framework: The study demonstrates that RAG is a viable approach for tackling complex tasks like SDG target detection in specialized domains, bridging the gap between general LLM capabilities and domain-specific knowledge.
- Retrieval Method Matters: The choice between lexical and dense retrieval significantly impacts performance, with lexical retrieval often performing better for EI tasks, especially with certain LLMs, while dense retrieval shows advantages for TD tasks.
- Prompt Engineering is Crucial: The "SDG-explicit" prompt generally leads to better results, highlighting the importance of clearly framing the task for the LLM. However, the "SDG-implicit" prompt offers insights into how LLMs generalize when the specific domain context is masked.
- LLM Choice Influences Outcomes: Different LLMs exhibit varying strengths. ChatGPT consistently performs well across many configurations, while ClimateGPT shows promise, particularly in recall for the TD task.
- Context Order Can Be Important: While not always the case, presenting retrieved passages in their ranked order often yields better results than a random order.
Key Results
- EI Task: ChatGPT combined with lexical retrieval and a single example in the prompt generally achieved the best performance across metrics and augmentation strategies.
- TD Task: ChatGPT achieved the highest precision when using densely retrieved passages. For recall, ClimateGPT-13b performed best with lexical retrieval.
- LLM Performance: ChatGPT was generally the top performer, followed by Llama2-13b-chat. ClimateGPT-13b showed competitive results, especially in recall for the TD task.
- Augmentation Impact: Having two examples in the prompt was beneficial for some LLMs (Llama2-13b-chat, ChatGPT) but detrimental to others (ClimateGPT-13b, Llama2-13b).
Limitations and Future Work
The authors acknowledge several limitations:
- Limited SDG Scope: The study focused on a subset of SDG targets, and a broader coverage could be explored.
- Geographical Bias: The dataset of reports is primarily from Europe, limiting the generalizability to other regions and environments.
- Annotation Subjectivity: While efforts were made for accurate annotation, the concept of "addressing an SDG target" can be subjective, leading to potential variations in judgment.
Future work directions include:
- Inter-task Retrofitting: Developing a method where the EI task informs the TD task, and vice-versa, to create a more integrated approach.
- Claim Detection Integration: Incorporating dedicated climate-aware claim detectors to improve passage selection.
- Automated Labeling and Fine-tuning: Using LLMs as assessors to automatically label larger datasets for further fine-tuning of models.
- Automatic Assessment of Correctness: Developing LLM-based methods to automatically evaluate the correctness of predicted results.
Summary updated