Climate Policy Transformer: Utilizing NLP to track the coherence of Climate Policy Documents in the Context of the Paris Agreement

Prashant Singh, Erik Lehmann, Mark TyrrellACL 2024

TL;DR

This paper introduces "CPo-CD," a novel dataset and fine-tuned NLP models for automatically tracking climate commitments within policy documents like NDCs and LTSs. By leveraging existing climate policy databases and advanced transformer models, the authors significantly streamline the analysis of these complex documents, making it easier to monitor global climate action.

What Problem Does This Paper Address?

The Paris Agreement relies on countries submitting Nationally Determined Contributions (NDCs) and Long-Term Strategies (LTS) to track progress towards climate goals. However, analyzing these documents is a monumental task due to their sheer volume, varied formats, and complex, often unstructured, content. Traditional methods are labor-intensive and time-consuming, hindering efficient monitoring and assessment of global climate commitments. This paper aims to bridge this gap by using Natural Language Processing (NLP) to automate and enhance the analysis of these critical policy documents.

Methods

The core of this work lies in applying deep learning-based NLP techniques, specifically transformer models, to classify climate policy documents. The authors fine-tune pre-trained transformer models on a custom dataset, "CPo-CD," to perform sequence classification. This classification aims to identify various components within policy texts, including:

  • Targets, Actions, Policies, and Plans (TAPP): Identifying the presence of these key policy elements.
  • Mitigation/Adaptation: Distinguishing between strategies aimed at reducing emissions versus those focused on adapting to climate change impacts.
  • Greenhouse Gas (GHG) Components: Specifically identifying targets related to GHG emissions.
  • Conditionality: Determining if commitments are dependent on external support or specific criteria.
  • Sectors: Assigning policy elements to specific economic or social sectors (e.g., Energy, Transport, Agriculture).

The methodology involves two main approaches: fine-tuning a generic large language model (LLM) embedding (BAAI/bge-base-en-v1.5) and fine-tuning a domain-specific LLM, ClimateBERT. For few-shot learning scenarios, they also utilize SetFit.

Datasets

The primary contribution is the creation of the CPo-CD (Climate Policy Classification Dataset). This dataset is built by agglomerating text extracts from two existing climate policy datasets:

  1. NDC Transport Tracker from the International Climate Initiative (IKI TraCS).
  2. NDC Sector Data from ClimateWatch.org.

The CPo-CD dataset is unique because it goes beyond simple text extracts. The authors retrieve the original source text from policy documents to provide richer context for the labeled passages. This process involves text processing, document retrieval, and context extraction to create a more robust dataset for training. The final CPo-CD dataset contains 13,728 samples for each sequence length, split into 12,538 for training and 1,190 for testing.

Key Insights and Ideas

  • The Power of Context: Simply using text snippets from existing datasets was insufficient. Retrieving and expanding these snippets with surrounding paragraphs from the original documents was crucial for effective classification, highlighting the importance of context in understanding policy intent.
  • Domain-Specific Models vs. Generic LLMs: While ClimateBERT, a domain-specific model, showed promise, the generic BAAI/bge-base-en-v1.5 model, when fine-tuned, often outperformed it, suggesting that advancements in general LLM architectures and scale can sometimes surpass specialized models, especially with sufficient fine-tuning.
  • TAPP Classification is Achievable: The models demonstrated good performance in identifying Targets and Actions, which are often more explicitly stated in policy documents.
  • Conditionality is Tricky: Classifying conditionality proved to be a significant challenge, likely due to the nuanced and often implicit nature of such statements within policy texts. This highlights a complex area for future NLP research.
  • Human Annotation as a Benchmark: Manual annotation by human reviewers provided a valuable benchmark, confirming the models' strong performance on 'Target' and 'Policy' categories but also revealing the inherent subjectivity and variability in human interpretation for 'Action' and 'Plans'.

Key Results

The fine-tuned models achieved commendable performance across various classification tasks:

  • TAPP Classification: The generic LLM (bge-base-en-v1.5) achieved F1 scores of 0.84 for 'Target' and 0.85 for 'Action', with slightly lower scores for 'Policy' (0.76) and 'Plan' (0.65). ClimateBERT performed comparably.
  • GHG Targets: The model achieved a high F1 score of 0.91 for GHG targets.
  • Mitigation/Adaptation: Both mitigation and adaptation were classified with high F1 scores of 0.92.
  • Sector Classification: Performance varied across sectors, with an overall F1 score of 0.76. Some sectors, like 'cross-cutting' and 'economy-wide', were more challenging.
  • Conditionality: Performance was lower, with F1 scores around 0.60-0.62, indicating the difficulty of this classification.
  • Carbon Emissions: The paper also tracked the carbon footprint of training, noting that fine-tuning the ClimateBERT base model for classification tasks accounted for less than 1% of its pre-training emissions.

Limitations and Future Work

The authors acknowledge several limitations:

  • Data Heterogeneity: The lack of a standardized reporting framework for climate policies across different countries and initiatives creates significant challenges for data harmonization and model scalability.
  • Linguistic Scope: The current models are primarily trained on English-language documents, limiting their applicability to other languages.
  • Document Type Diversity: The focus on NDCs and LTSs means the models do not yet cover other crucial policy documents like local policies and laws.
  • Conditionality Complexity: The nuanced nature of conditionality clauses makes accurate classification difficult, suggesting that generative language models with advanced reasoning capabilities might be needed.
  • Dataset Limitations: Despite efforts to enrich the data, some areas of the training data may still suffer from quality issues due to the original databases' limitations and the statistical matching process.

Future work could involve expanding the models to include multilingual capabilities, incorporating a broader spectrum of document types, and exploring more advanced NLP architectures to better handle complex classifications like conditionality.

Summary updated

BibTeX
@inproceedings{singh2024climate,
  author = {Prashant Singh and Erik Lehmann and Mark Tyrrell},
  title = {{Climate Policy Transformer: Utilizing NLP to track the coherence of Climate Policy Documents in the Context of the Paris Agreement}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure