Abstract
Environmental, Social, and Governance (ESG) reporting is increasingly susceptible to "greenwashing," where companies use vague or exaggerated claims to obscure actual sustainability performance. This paper presents A3CG, a novel dataset and framework designed to enhance the robustness of ESG analysis. By transitioning from simple topic detection to Aspect-Action Analysis, we enable the differentiation between concrete commitments and non-committal rhetoric. We evaluate various learning paradigms, including contrastive learning and large language models (LLMs), on their ability to generalize across unseen sustainability categories. Our findings indicate that while specialized supervised models achieve peak performance on known topics, LLMs demonstrate greater stability when encountering new ESG domains.
Introduction
As stakeholders demand greater transparency in corporate sustainability, the prevalence of greenwashing has risen significantly. Traditional NLP tools often fail to distinguish between a company's actual achievements and its aspirational or evasive statements. For instance, a model might interpret any mention of "carbon reduction" as a positive indicator, regardless of whether the report describes a completed project or a vague future possibility. To address this, we propose the Aspect-Action Analysis with Cross-Category Generalization (A3CG) task.
The Aspect-Action Framework
The core of the A3CG framework is the extraction of (Aspect, Action) pairs from corporate reports. Unlike standard aspect-based sentiment analysis, our framework categorizes the "Action" into three distinct levels of commitment:
- Implemented: Actions that have already been completed or are currently in progress.
- Planning: Concrete commitments to future actions.
- Indeterminate: Vague, hedged, or non-committal language (e.g., "we may consider," "where possible").
This "Indeterminate" label serves as a critical flag for potential greenwashing, highlighting areas where corporate rhetoric lacks substance.
Methodology and Dataset
The A3CG dataset comprises 2,004 manually annotated statements from 1,679 sustainability reports listed on the Singapore Exchange (SGX) between 2017 and 2022. The data covers 10 diverse sustainability categories. To test the models' ability to handle the evolving nature of ESG reporting, we utilize a cross-category evaluation setup where models are trained on one set of categories and tested on entirely different, "unseen" ones.
We explored several learning paradigms:
- Supervised Fine-tuning: Using models like T5 and BERT.
- Contrastive Learning (CL): Helping models learn the underlying structure of ESG claims by grouping similar categories.
- Adversarial Learning (AL): Encouraging models to focus on universal action descriptions rather than category-specific jargon.
- Large Language Models (LLMs): Evaluating zero-shot and few-shot capabilities of models like GPT-4o and Claude 3.5.
Key Results
Our experiments reveal a trade-off between absolute performance and generalization stability. Specialized supervised models, such as GRACE, achieved the highest F1 scores on unseen categories (47.51). However, these "specialists" often experience significant performance degradation when the topic shifts. In contrast, LLMs like Claude 3.5 Sonnet act as "generalists"; while their peak scores are slightly lower, they maintain more consistent performance across diverse categories.
A significant finding is that identifying the Action (the level of commitment) is consistently more challenging than identifying the Aspect (the topic). Supervised models often struggle with complex grammatical structures, while LLMs, despite their linguistic prowess, sometimes fail to grasp the subtle pragmatic intent behind evasive corporate language.
Discussion and Insights
- The Power of Contrast: Contrastive Learning proved highly effective for generalization. For the BERT-ST model, adding CL improved performance on unseen categories by nearly 12 points.
- Environmental Bias in LLMs: LLMs showed a higher recall for environmental topics compared to social or governance ones, likely reflecting a bias in their pre-training data which often equates "sustainability" primarily with environmental issues.
- Greenwashing Detection: By explicitly labeling "Indeterminate" actions, A3CG provides investors and regulators with a transparent tool to audit corporate claims. A high frequency of indeterminate labels in a report serves as a data-driven red flag.
Conclusion and Future Work
A3CG represents a significant step toward automated, robust greenwashing detection. Future research will focus on expanding the dataset to include multi-lingual reports and integrating the framework with external data sources for real-world verification of corporate claims. Scaling the dataset and exploring the fine-tuning of smaller, open-source LLMs also remain promising avenues for closing the generalization gap in ESG analysis.