TL;DR
This paper introduces "Promise Verification," a new task to automatically assess the credibility of corporate promises, particularly those related to Environmental, Social, and Governance (ESG) initiatives. They've created ML-Promise, the first multilingual dataset for this task, and demonstrated that Retrieval-Augmented Generation (RAG) models can effectively identify promises, find supporting evidence, and evaluate their clarity and verification timelines.
What Problem Does This Paper Address?
In today's world, corporate promises about sustainability, social responsibility, and governance significantly influence public trust and company reputation. However, the sheer volume and complexity of these commitments, coupled with the rise of "greenwashing" (misleading claims about environmental efforts), make it difficult to verify their authenticity. Existing methods for analyzing corporate reports often lack the focus on promise verification, especially across different languages and considering multimodal (text and image) information. This paper aims to bridge this gap by providing a systematic approach and a valuable resource for evaluating the integrity of corporate ESG promises.
Methods
The core of this work is the development of a Promise Verification framework, which involves four key steps:
- Promise Identification (PI): Determining if a statement is a corporate promise.
- Actionable Evidence (AE): Identifying if supporting evidence for the promise exists within the report.
- Clarity of the Promise-Evidence Pair (CPEP): Assessing how clearly the evidence backs up the promise.
- Timing for Verification (TV): Determining when the promise's fulfillment can be checked.
To tackle these tasks, the authors leverage Retrieval-Augmented Generation (RAG). This approach combines a retrieval mechanism (finding similar examples from a training set) with a large language model (LLM, specifically GPT-40) to generate more accurate and contextually relevant outputs. They also explore multimodal approaches, using image inputs from PDF reports alongside text, to capture a more comprehensive view of corporate disclosures.
Datasets
The paper introduces the ML-Promise dataset, a novel benchmark for promise verification. It comprises 3,010 annotated instances (2,010 for training, 1,000 for testing) across five languages: Chinese, English, French, Japanese, and Korean. The dataset is curated from ESG reports of companies across various industries and countries, aiming for linguistic and contextual diversity.
Key Insights and Ideas
- The Rise of Promise Verification: The paper argues for a dedicated task to combat greenwashing and ensure corporate accountability, moving beyond general ESG report analysis.
- RAG's Effectiveness: RAG significantly improves performance, particularly in assessing the clarity of promise-evidence pairs and predicting verification timelines, across multiple languages.
- Multimodality Matters: For certain tasks (Promise Identification and Actionable Evidence), incorporating image-based analysis from PDF reports can be beneficial, especially for regions with visually rich reporting styles.
- Language and Regional Nuances: ESG reporting styles and the clarity of promises vary significantly across countries and languages, underscoring the need for multilingual and culturally aware analysis. For instance, Taiwanese and Korean companies tend to make more short-term promises.
- Market Cap and Promise Articulation: Larger companies are more likely to make explicit ESG promises, likely due to increased regulatory pressure and resources for dedicated sustainability teams.
- Industry-Specific Trends: Different industries have distinct patterns in promise-making and evidence provision, influenced by regulation, public scrutiny, and business models (e.g., highly regulated biomedical sectors focus on precision, while energy sectors emphasize transition).
Key Results
- RAG Performance: RAG generally improves F1 scores across tasks and languages. For example, in English, RAG boosted performance for all tasks, and in Chinese, it enhanced clarity and timing prediction.
- Multimodal vs. Text: While RAG with text input was superior for clarity and timing, image input with RAG showed promise for promise identification and actionable evidence, especially in Chinese and Korean reports.
- Inter-annotator Agreement: The study achieved substantial agreement (Cohen's Kappa > 0.60) across all annotation tasks, indicating reliable annotations.
- Promise Clarity: Around 35-40% of evidence was rated "not clear" across languages, highlighting the need for better promise verification.
- Misleading Evidence: Approximately 4% of instances contained potentially misleading evidence in English and Japanese datasets.
- SemEval-2025 PromiseEval: The proposed RAG-based system achieved top rankings in Korean and Japanese, and second place in English, French, and Chinese, demonstrating its competitiveness.
Limitations and Future Work
- Dataset Scope: The ML-Promise dataset, while multilingual, is limited to a few countries and may not fully represent global corporate promise communication styles. Future work should expand the dataset to more regions and languages.
- RAG Optimization: RAG's performance can be inconsistent across languages and tasks, suggesting a need for further optimization and task-specific adjustments, especially for imbalanced datasets.
- Informal Commitments: The current dataset focuses on well-documented promises; less publicized or informal commitments are not included. Future work could explore incorporating evidence from a wider range of sources.
- Longitudinal Tracking: The current study focuses on single reports. Future work aims to extend the framework to annual compliance checking and longitudinal tracking of promise-related actions over multiple years.
- Multimodal Refinement: Further exploration of multimodal approaches and balancing text and image inputs for different tasks is recommended.
Summary updated