An NLP Benchmark Dataset for Assessing Corporate Climate Policy Engagement

Gaku Morio, Christopher D ManningNeurIPS 2023 · OpenReview

As "greenwashing" becomes a major concern for investors, regulators, and consumers, verifying whether a company’s public environmental claims match its actual lobbying and policy engagement is critical. However, monitoring this at scale is a massive undertaking. It traditionally requires domain experts to manually sift through hundreds of pages of sustainability reports, financial disclosures, and regulatory filings.

While Natural Language Processing (NLP) offers a way to automate this, existing datasets in the climate domain are often too small, lack diversity, or fail to provide the specific evidence needed for high-stakes financial or policy decisions. To address this, researchers have introduced a large-scale NLP benchmark dataset containing over 10,000 documents designed to track how corporations actually engage with climate policy.

Sourcing the Data: The LobbyMap Platform

The foundation of this new benchmark is data from LobbyMap, a platform operated by the global think tank InfluenceMap. LobbyMap provides human-expert assessments of corporate climate lobbying, categorizing engagements and assigning stances based on specific evidence found in corporate documents.

Example of human-curated evidence from the LobbyMap platform, which serves as the ground truth for the dataset.

The resulting dataset is massive and diverse:

  • Scale: Over 10,000 documents, 28 million words, and 60,000 pages.
  • Coverage: Focuses on companies in the CA100+ (Climate Action 100+) investor initiative, spanning 14 sectors (e.g., automobiles, energy) and headquartered across all continents except Antarctica.
  • Format: Provided in JSON Lines format, including sentence-tokenized text, query/stance triplets, and metadata.

Methodology: From Raw PDFs to a Structured NLP Task

To convert raw corporate PDFs and LobbyMap assessments into a structured NLP dataset, the researchers developed a sophisticated pipeline.

Task formulation: Extracting corporate climate policy engagement triplets (Query, Stance, Page indices) from PDF documents.
  1. Robust Text Extraction: Because corporate reports vary wildly in layout and often rely heavily on images, the team used three different parsers—Fitz (PyMuPDF) for digital text, alongside docTR and Tesseract for OCR-based extraction.
  2. Evidence Alignment: Using NLTK and partial string matching, they aligned the human-curated "evidence snippets" from LobbyMap back to specific pages in the original PDFs.
  3. Modeling Pipeline: The problem was framed as a three-stage, page-wise classification task:
  4. Evidence Page Detector: Does this page contain relevant evidence?
  5. Query Classifier: Which of the 13 climate policy categories (e.g., Carbon Tax, Renewable Energy) is being discussed?
  6. Stance Classifier: What is the company's position on a 5-level scale (from "Strongly Supporting" to "Opposing")?
  7. Supplementary Task: The authors also experimented with Flan-T5 to generate natural language "scrutiny comments" explaining why a specific stance was assigned.

Key Insights and Model Performance

The benchmark reveals several critical challenges in applying NLP to corporate climate documents:

  • The "Long Document" Challenge: Most corporate reports average 6–9 pages, but many exceed 100 pages. Model performance (measured by Strict F-score) significantly declines as document length increases, highlighting the difficulty of maintaining context across long sequences.
  • Domain Adaptation is Essential: ClimateBERT, a model pre-trained specifically on climate-related text, consistently outperformed standard BERT. This proves that specialized vocabulary is vital for this domain.
  • The Localization Gap: There is a massive gap between "Document-level" F-scores (~74%) and "Strict" F-scores (~53%). Models are relatively good at guessing if a company supports a policy somewhere in a report, but struggle to pinpoint the exact page and sentence that proves it.
  • Nuance is Hard: Models frequently confused a "Supporting" stance with "No position/Mixed," suggesting that the subtle, hedging language typical of corporate reports remains a major hurdle.

When benchmarking several models, Longformer-large emerged as the top performer due to its ability to handle longer context windows:

Metric (Strict) P (Page Index) Q (Query) S (Stance)
Most-frequent Baseline 41.2% 19.6% 17.5%
BERT-base 50.2% 31.9% 25.8%
ClimateBERT 50.2% 32.2% 26.8%
Longformer-large 53.6% 38.7% 31.5%

In the supplementary task, Flan-T5-XL achieved a ROUGE-L score of 35.6 for comment generation, indicating a promising start for automated, analyst-style explanations.

Limitations and Future Work

While this dataset bridges a crucial gap between NLP and climate change research, several challenges remain:

  • No "Negative Samples": The current dataset only includes documents containing at least one piece of evidence. Real-world applications must also handle "null" documents with no relevant climate information.
  • Page-Wise vs. Global Context: The pipeline processes pages independently. Future models should evaluate documents holistically to resolve contradictions between different sections.
  • Multi-modality: Corporate reports are rich with charts, tables, and infographics. This text-only dataset leaves a wealth of visual evidence untapped.
  • Language: The dataset is currently limited to English, despite corporate climate engagement being a global, multilingual issue.

This benchmark provides a vital stepping stone for developing AI tools capable of holding corporations accountable for their climate policy engagements.

Summary updated

Abstract

As societal awareness of climate change grows, corporate climate policy engagements are attracting attention. We propose a dataset to estimate corporate climate policy engagement from various PDF-formatted documents. Our dataset comes from LobbyMap (a platform operated by global think tank InfluenceMap) that provides engagement categories and stances on the documents. To convert the LobbyMap data into the structured dataset, we developed a pipeline using text extraction and OCR. Our contributions are: (i) Building an NLP dataset including 10K documents on corporate climate policy engagement. (ii) Analyzing the properties and challenges of the dataset. (iii) Providing experiments for the dataset using pre-trained language models. The results show that while Longformer outperforms baselines and other pre-trained models, there is still room for significant improvement. We hope our work begins to bridge research on NLP and climate change.
BibTeX
@inproceedings{morio2023nlp,
  author = {Gaku Morio and Christopher D Manning},
  title = {{An NLP Benchmark Dataset for Assessing Corporate Climate Policy Engagement}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=GF5l0F19Bt},
}
Figure