TL;DR
This paper tackles the challenge of extracting structured sustainability information from complex corporate reports, particularly for the new European Sustainability Reporting Standard (ESRS). They achieve this by extending an existing ontology to cover ESRS topical standards and then using Large Language Models (LLMs) guided by this ontology to automatically generate Knowledge Graphs (KGs) from these reports. This approach promises to make sustainability data more transparent, reusable, and easier to analyze.
What Problem Does This Paper Address?
The increasing demand for corporate transparency in environmental, social, and governance (ESG) matters has led to new, comprehensive reporting standards like the European Sustainability Reporting Standard (ESRS). However, implementing these standards is a significant challenge for companies. It requires sifting through vast amounts of textual information from various sources to identify and report specific data points. This manual process is time-consuming and prone to errors. Existing ESG frameworks like GRI and SASB are being superseded by ESRS, which demands even more detailed information, especially regarding financial materiality analysis of climate-related risks and opportunities. The paper aims to automate the extraction and structuring of this crucial climate change-related information from non-financial reports to facilitate the transition to ESRS.
Methods
The core of the paper's methodology lies in combining ontologies and Large Language Models (LLMs) for knowledge extraction.
- Ontology Extension: They start with an existing ontology, OntoSustain, which already covers some sustainability indicators. They extend this ontology to specifically model the ESRS 2 Topical Standard for environmental disclosures. This extension focuses on the "Climate Change" (E1) category and defines key aspects like Impact, Risks, Opportunities, Strategy, Actions, and Targets.
- Knowledge Graph (KG) Construction: The process leverages an LLM (GPT-4) guided by the extended ontology.
- Chunking: Large disclosure reports are first broken down into smaller, manageable chunks using NLTKTextSplitter for content-aware segmentation.
- Prompt Generation: The ontology's concepts and relations are used to automatically generate prompts for the LLM. These prompts include task descriptions, relevant context from the ontology, and few-shot examples to guide the LLM's extraction process.
- Knowledge Extraction: The LLM processes these chunks and prompts to extract relevant information, aiming to identify concepts and their relationships as defined by the ontology.
- KG Generation: The extracted triples (subject-predicate-object) are then used to construct a Knowledge Graph, providing a structured representation of the sustainability information.
- Summarization and Voting: For certain information like organization name and reporting date, majority voting is used to consolidate findings.
Datasets
The study utilized a dataset of 124 pre-processed sustainability reports, primarily from North American companies, originally provided by Bronzini et al. (2023). However, for their specific focus on ESRS, they filtered this down to reports from 14 EU-based companies. These companies represent diverse GICS sectors and their reports were mostly from the 2021 fiscal year, with a few from 2020 and 2017. No new benchmark datasets were introduced; instead, they focused on applying their method to existing corporate reports.
Key Insights and Ideas
- Ontology-Guided LLMs are Powerful for Structured Extraction: The paper demonstrates that by providing LLMs with a structured ontological framework and relevant examples, they can effectively extract complex, domain-specific information from unstructured text. This moves beyond simple keyword extraction to understanding relationships and context.
- ESRS Demands Granular Detail: The transition from GRI to ESRS requires more detailed information, particularly concerning the financial materiality of climate-related risks and opportunities. This highlights the need for sophisticated tools to manage this increased reporting burden.
- LLM Hallucinations are a Real Concern, but Mitigatable: The authors acknowledge the potential for LLMs to "hallucinate" or generate incorrect information. They address this through careful prompt engineering, providing clear examples, and relying on expert review for validation.
- Certain Aspects are Easier to Extract than Others: The evaluation revealed that aspects like "Strategy" were consistently well-matched by annotators, while "Impact" and "Opportunities" proved more challenging, often being confused with "Risks" or "Actions." This suggests that the definitions and examples for these aspects might need further refinement.
Key Results
- Successful KG Construction: The proposed method successfully extracted and structured information from corporate sustainability reports into Knowledge Graphs, guided by the extended ESRS ontology.
- High Match Rate for Key Aspects: The "Strategy" aspect achieved a perfect match rate across all annotators. "Risks," "Actions," and "Targets" also showed high agreement.
- Moderate Inter-Annotator Agreement: The study achieved a Cohen's Kappa score of 0.512, indicating a moderate level of agreement among human annotators when evaluating the extracted topical matches. This suggests the task has some inherent ambiguity or complexity.
- Identification of Vague Descriptions: The evaluation highlighted that aspects with lower topical match scores often contained vaguer descriptions, particularly for "Targets" (lack of quantifiable goals) and "Opportunities" (overly general definitions).
Limitations and Future Work
Limitations:
- Computational Cost: Processing large documents and passing each chunk with the ontology to the LLM is computationally intensive and time-consuming, posing a challenge for smaller research labs.
- Ontology Scope: The extended ontology currently focuses on ESRS 2 Topical Standard for Environmental Standards (specifically E1 - Climate Change). It requires further evaluation and extension to cover other Topical Standards (Social, Governance) and a broader range of environmental topics.
- KG Validation: The generated KGs need more extensive validation by domain experts to ensure consistency and completeness.
- Human-in-the-Loop: The authors suggest that a human-in-the-loop approach, where experts customize examples or verify extracted data, could significantly improve accuracy and address LLM limitations.
Future Work:
- Develop Better Chunking Techniques: Explore more efficient pre-processing and chunking methods to reduce processing time.
- Expand Ontology Coverage: Extend the ontology to encompass a wider range of ESRS standards and environmental, social, and governance (ESG) sectors.
- Enhance KG Validation: Implement more robust KG validation processes, potentially involving more domain experts.
- Refine Prompt Engineering: Improve prompt design, especially for challenging aspects like "Impact" and "Opportunities," by providing more descriptive and specific examples.
- Address LLM Biases: Investigate and mitigate potential biases in LLMs that could perpetuate biases present in the source data.
- Transparency and Accountability: Ensure the processes and algorithms used are transparent and auditable to foster trust in automated sustainability reporting.
Summary updated