As "greenwashing" becomes a major concern for investors, regulators, and consumers, verifying whether a company’s public environmental claims match its actual lobbying and policy engagement is critical. However, monitoring this at scale is a massive undertaking. It traditionally requires domain experts to manually sift through hundreds of pages of sustainability reports, financial disclosures, and regulatory filings.
While Natural Language Processing (NLP) offers a way to automate this, existing datasets in the climate domain are often too small, lack diversity, or fail to provide the specific evidence needed for high-stakes financial or policy decisions. To address this, researchers have introduced a large-scale NLP benchmark dataset containing over 10,000 documents designed to track how corporations actually engage with climate policy.
Sourcing the Data: The LobbyMap Platform
The foundation of this new benchmark is data from LobbyMap, a platform operated by the global think tank InfluenceMap. LobbyMap provides human-expert assessments of corporate climate lobbying, categorizing engagements and assigning stances based on specific evidence found in corporate documents.
The resulting dataset is massive and diverse:
- Scale: Over 10,000 documents, 28 million words, and 60,000 pages.
- Coverage: Focuses on companies in the CA100+ (Climate Action 100+) investor initiative, spanning 14 sectors (e.g., automobiles, energy) and headquartered across all continents except Antarctica.
- Format: Provided in JSON Lines format, including sentence-tokenized text, query/stance triplets, and metadata.
Methodology: From Raw PDFs to a Structured NLP Task
To convert raw corporate PDFs and LobbyMap assessments into a structured NLP dataset, the researchers developed a sophisticated pipeline.
- Robust Text Extraction: Because corporate reports vary wildly in layout and often rely heavily on images, the team used three different parsers—Fitz (PyMuPDF) for digital text, alongside docTR and Tesseract for OCR-based extraction.
- Evidence Alignment: Using NLTK and partial string matching, they aligned the human-curated "evidence snippets" from LobbyMap back to specific pages in the original PDFs.
- Modeling Pipeline: The problem was framed as a three-stage, page-wise classification task:
- Evidence Page Detector: Does this page contain relevant evidence?
- Query Classifier: Which of the 13 climate policy categories (e.g., Carbon Tax, Renewable Energy) is being discussed?
- Stance Classifier: What is the company's position on a 5-level scale (from "Strongly Supporting" to "Opposing")?
- Supplementary Task: The authors also experimented with Flan-T5 to generate natural language "scrutiny comments" explaining why a specific stance was assigned.
Key Insights and Model Performance
The benchmark reveals several critical challenges in applying NLP to corporate climate documents:
- The "Long Document" Challenge: Most corporate reports average 6–9 pages, but many exceed 100 pages. Model performance (measured by Strict F-score) significantly declines as document length increases, highlighting the difficulty of maintaining context across long sequences.
- Domain Adaptation is Essential: ClimateBERT, a model pre-trained specifically on climate-related text, consistently outperformed standard BERT. This proves that specialized vocabulary is vital for this domain.
- The Localization Gap: There is a massive gap between "Document-level" F-scores (~74%) and "Strict" F-scores (~53%). Models are relatively good at guessing if a company supports a policy somewhere in a report, but struggle to pinpoint the exact page and sentence that proves it.
- Nuance is Hard: Models frequently confused a "Supporting" stance with "No position/Mixed," suggesting that the subtle, hedging language typical of corporate reports remains a major hurdle.
When benchmarking several models, Longformer-large emerged as the top performer due to its ability to handle longer context windows:
| Metric (Strict) | P (Page Index) | Q (Query) | S (Stance) |
|---|---|---|---|
| Most-frequent Baseline | 41.2% | 19.6% | 17.5% |
| BERT-base | 50.2% | 31.9% | 25.8% |
| ClimateBERT | 50.2% | 32.2% | 26.8% |
| Longformer-large | 53.6% | 38.7% | 31.5% |
In the supplementary task, Flan-T5-XL achieved a ROUGE-L score of 35.6 for comment generation, indicating a promising start for automated, analyst-style explanations.
Limitations and Future Work
While this dataset bridges a crucial gap between NLP and climate change research, several challenges remain:
- No "Negative Samples": The current dataset only includes documents containing at least one piece of evidence. Real-world applications must also handle "null" documents with no relevant climate information.
- Page-Wise vs. Global Context: The pipeline processes pages independently. Future models should evaluate documents holistically to resolve contradictions between different sections.
- Multi-modality: Corporate reports are rich with charts, tables, and infographics. This text-only dataset leaves a wealth of visual evidence untapped.
- Language: The dataset is currently limited to English, despite corporate climate engagement being a global, multilingual issue.
This benchmark provides a vital stepping stone for developing AI tools capable of holding corporations accountable for their climate policy engagements.
Summary updated