Commitment Status for Climate Policy Text: Clinical Assertion Detection Against Greenwashing

Clinical language processing learned long ago that finding a disease name in a note is not the same as knowing the patient has it. "Denies chest pain," "cannot rule out pneumonia," and "history of MI" all mention a condition. Only one of them asserts an active condition, and good clinical extractors label each mention with its status: present, absent, uncertain, or historical. Text analysis of climate and sustainability policy has mostly not made this step. Municipal climate plans, corporate climate disclosures, and annual reports are mined for which policies or targets are mentioned. Whether a policy is enacted, funded, merely aspirational, conditional, or abandoned is usually left out. This matters for two reasons. Econometric studies that turn report text into policy indices count "we discontinued our offset program" the same as "we expanded it." And when language models score disclosures, a company can raise its score by rewording intentions as achievements without changing any fact.

We propose a commitment-status schema for policy text, adapted from clinical assertion labels. Its key design choice is that a strong status such as "achieved" or "funded" may be assigned only when a specific type of verbatim evidence is present: a measured value with a baseline year, or a budget figure with a funding source. Verb tense or tone of confidence does not count. The central test is whether status labels assigned this way stay put when disclosures are rewritten to sound more accomplished without new facts, while LLM judges' scores move. A second test asks whether conditioning on status changes substantive findings that published policy analyses drew from mention counts.

Core Claim

Grounding status in evidence type rather than phrasing makes extracted commitment status nearly invariant to framing-only greenwashing. This holds whether the extractor is a large prompted model or a small fine-tuned classifier. The contribution is therefore the schema, its evidence rules, an annotated benchmark, and the invariance and downstream tests. It is not a particular prompting pipeline.

The motivating evidence comes from Judging It, Washing It, which scored 1,416 European responses to the 2022 Carbon Disclosure Project (CDP) climate questionnaire with GPT-4o-mini. Their "fixed accuracy" rewrite forbade fabricated claims but allowed aspirational intentions to be recast as achievements. It still raised scores by +0.56 on a 5-point scale and +11 on a 100-point pairwise scale. That manipulation is, in clinical terms, a status flip: "planned" relabeled as "present."

Research Program

The schema has seven statuses for each extracted commitment. Achieved means a measured outcome against a stated baseline. Funded means an allocated budget or named financing instrument. Scheduled means a dated implementation step with an owner. Aspirational means a goal without means. Conditional means dependent on external funding or regulation. Discontinued means withdrawn or replaced. Historical means describing a past state, not a commitment.

Each strong status carries an evidence rule. Achieved requires a quoted span containing a quantity and a reference point. Funded requires a quoted monetary amount or instrument. If the required span is missing, the label is capped at aspirational or scheduled. This mirrors how CheXbert (Combining Automatic Labelers and Expert Annotations) attaches a positive/negative/uncertain/blank state to every finding. It also mirrors the evidence-grounding and pruning steps of Self-Verification Improves Few-Shot Clinical Information Extraction, where the model must quote a supporting span and discard negated or historical items. We borrow the verification idea from that paper. We do not borrow its open-ended omission search, because policy discovery is a separate problem.

We will annotate two corpora. The first is a stratified sample of roughly 60 of the 227 California Climate Action Plans parsed in Challenges in End-to-End Policy Extraction from Climate Action Plans. We start from their extracted policy names, because that paper shows discovery from scratch collapses (Llama 2 7B entity F1 = 15.48), while attribute extraction given gold names reaches F1 = 64.95. Status assignment is an attribute-level task, so it sits on the part of the pipeline that works. The second corpus is the CDP 2022 responses to the absolute and intensity target questions (Q4.1a/b). Two annotators with policy backgrounds label status and evidence spans, and we report agreement.

We will build three extractors of increasing cost. The first is a small encoder with per-commitment status heads, trained CheXbert-style: weak labels from a rule system first, then fine-tuning on the expert set. The second is a prompted large model that applies the evidence rules in two phases. It first verifies whether the required evidence span exists, then assigns status. This adapts the two-phase leaf verification of mCODEGPT, which reported 93.9% accuracy and 1.1% hallucination on synthetic oncology notes. The third is the same large model without the evidence rules.

Baselines

The naive reference point is the mention-count approach that current econometric work uses. It applies seed-word expansion and TF-IDF (term frequency weighted by rarity across documents) to policy terms, as in The Impact of High-Tech Policies on Corporate ESG Performance. Every mention is scored as an active commitment. This baseline is beaten if status-weighted indices differ materially from it on documents where annotators mark many mentions as aspirational or discontinued. It has no status output, so it serves as the reference for the downstream comparison.

The strongest existing scoring method is the LLM-as-judge setup of Judging It, Washing It. It uses numerical ratings with token-probability averaging and pairwise win rates. The reported fixed-accuracy inflation is +0.56 (5-point) and +11 (100-point). We will rerun their protocol with the same evaluator family on our regenerated rewrites, so both systems see identical text. To beat it, a status-derived disclosure score must shift less than one-quarter as much under fixed-accuracy rewriting. It must also separate CDP A-List from non-A-List firms at least as well as the judge does on original text. That paper reports a correlation of r² = 0.70 between its two scoring modes but gives no A-List separation metric, so we will compute that metric for both systems ourselves.

The strongest extraction competitor is single-pass prompting with status as an extra output field. It uses the same large model, the same few-shot examples, and the same token budget, matched within 20% of the evidence-rule variant. Self-verification is the published improvement over single-pass extraction: GPT-4 F1 rose from 0.419 to 0.530 on EBM-NLP trial arms and from 0.487 to 0.533 on MIMIC-IV ICD-10 codes (Self-Verification). However, no reported result was found on Lacuna for commitment-status classification on policy text. Our status macro-F1 numbers will therefore be the first on these corpora.

The ablation that isolates the key ingredient is the same evidence-grounded pipeline with the evidence-type rules removed. The model still quotes a span, but any span may justify any status. If invariance survives without the rules, the rules are not what does the work.

Why not just…?

Why not just add a status field to a single LLM prompt? This is the most serious threat. Large models already handle negation and tense reasonably well. Our expectation is that they assign status largely from phrasing, and phrasing is exactly what greenwashing changes. The single-pass baseline and the no-rules ablation are evaluated on the same rewritten disclosures. If either matches the full system's invariance, we will report that the evidence rules are unnecessary and simplify the method accordingly.

Why not just use a negation-cue dictionary on top of keyword counts? A dictionary approach, in the spirit of early clinical rule systems and the cascade of A cascade of classifiers for extracting medication information, can catch "discontinued" or "no longer." It cannot detect that "we have embarked on reducing emissions" lacks a measured outcome. In radiology, rules plateaued below learned models (CheXpert F1 0.743 vs. CheXbert 0.798). We include this rule system as the weak labeler and as a baseline, so its ceiling on policy text is measured directly.

Why not just fine-tune a small classifier and skip large models? This could work as well, and the proposal is built to accept that outcome. The small encoder is one of our three extractors. Cross-encoders lagged GPT-4-Turbo on taxonomy alignment of climate pledges (MiniCDP 30.6% vs. 49.4% on the balanced set in Aligning Unstructured Paris Agreement Climate Plans). Status assignment with explicit evidence rules is a narrower task than open taxonomy mapping, though. If the small model matches the large one on status F1 and invariance, it becomes the recommended deployable system, and the contribution rests on the schema and tests.

Why not just tell the judge to ignore framing? An instruction does not give the judge a checkable criterion. We will include an "evidence-only" judge prompt as a variant of the judge baseline. Its inflation under rewriting will show whether instructions alone close the gap.

Proposed Experiments

The first experiment establishes extraction quality. We report status macro-F1 and evidence-span exact and overlap match on the annotated CAP and CDP sets for all three extractors and the rule baseline, alongside inter-annotator agreement as a ceiling.

The decisive invariance experiment regenerates unconstrained, fixed-accuracy, and fixed-accuracy-plus-length rewrites of the CDP responses with the protocol of Judging It, Washing It. For each system we measure the fraction of commitments whose status changes and the shift in the derived disclosure score. We expect fixed-accuracy rewrites to leave evidence-ruled status almost unchanged, because they cannot add quantities or budgets. Unconstrained rewrites can fabricate numbers, so they will defeat any text-only method. We will report that failure honestly as the limit of the approach. If the rule-based status shift under fixed-accuracy rewriting is not clearly below the judge's, the core claim fails.

The downstream experiment revisits a substantive finding of the CAP study: per-capita income correlates positively with the density of adaptation policies. We re-estimate that correlation counting only funded, scheduled, or achieved adaptation policies, then compare it with the all-mentions estimate. If richer jurisdictions mainly write more aspirational policies, the relationship should weaken. Either outcome is informative about what mention-based policy measures capture.

As a smaller transfer check, we apply the status-weighted index and the TF-IDF index to a subset of annual report sections and compare the rank correlation of firm scores. This tests whether mention-count indices of the kind used in The Impact of High-Tech Policies on Corporate ESG Performance are dominated by aspirational language.