EnClaim: A Style Augmented Transformer Architecture for Environmental Claim Detection

Diya Saha, Manjira Sinha, Tirthankar DasguptaACL 2024

TL;DR

This paper introduces EnClaim, a novel transformer-based architecture that enhances environmental claim detection by incorporating stylistic features alongside linguistic ones. By analyzing linguistic patterns and stylistic nuances like vagueness and conviction, EnClaim significantly improves the accuracy of identifying greenwashing and environmental claims, outperforming existing models and even large language models in certain scenarios.

What Problem Does This Paper Address?

The growing global focus on sustainability has unfortunately led to an increase in "greenwashing" – companies making exaggerated or misleading claims about their environmental commitments. This practice can deceive consumers and hinder genuine environmental progress. While automated methods are needed to detect these claims at scale, existing approaches often overlook the subtle linguistic and stylistic cues that can differentiate genuine claims from mere marketing statements. This paper addresses the critical need for a more robust system to automatically detect and validate environmental claims by considering these often-overlooked stylistic elements.

Methods

The core of EnClaim is a transformer-based architecture, specifically leveraging BERT. What makes EnClaim novel is its augmentation with stylistic features. These features are derived from:

  • LIWC (Linguistic Inquiry and Word Count): This tool analyzes text for frequencies of words falling into over 80 linguistic, psychological, and topical categories, providing a 73-dimensional vector representing these linguistic aspects.
  • Three Key Stylistic Factors:
    • Vagueness: Assessed using a tree-BiLSTM model to quantify the lack of explicit details or presence of generalizations. The score is inverted so higher scores indicate clearer text.
  • Conviction: Measured using the Empath library to detect the presence of pride and trust, and the absence of timidity and nervousness, indicating a strong authorial stance.
  • Commitment: Evaluated using the Empath library to identify optimism, zest, and achievement, reflecting the writer's dedication to a particular point.

These stylistic features (3 dimensions) are concatenated with the LIWC features (73 dimensions) to form a 76-dimensional "styled vector." This styled vector is then combined with the BERT pooler output (768 dimensions) to create an 844-dimensional input for a final dense layer that predicts the claim score.

Datasets

The paper utilizes three datasets for training and evaluation:

  • Dataset-ECD (Environmental Claim Detection Dataset): Contains text from sustainability reports, earning calls, and annual reports of listed companies, annotated for environmental claims. It has 2647 examples.
  • Dataset-GCC (Green-Claims Corpus): Consists of 773 tweets from the cosmetics and electronics domains, classified as "green-claim" or "not green-claim."
  • Dataset-SCDC (Scientific Claim Detection Corpus): A dataset of biomedical paper abstracts with claims labeled on a continuous scale (False to True). For binary classification, scores >= 0.5 are considered claims. This dataset is used to test generalizability outside the environmental domain.

Key Insights and Ideas

  • Stylistic Features Matter: The paper strongly suggests that beyond just the content of a statement, its stylistic presentation (how it's said) is crucial for identifying environmental claims. Vagueness, conviction, and commitment are not just linguistic quirks but important indicators.
  • Synergy of Context and Style: Combining deep language understanding (BERT) with stylistic analysis provides a more comprehensive approach to claim detection than either method alone.
  • Generalizability: EnClaim shows promise in generalizing to different domains, as demonstrated by its performance on the SCDC dataset, which is not strictly environmental.
  • LLMs Aren't Always Superior: While LLMs like LLAMA-2 are powerful, EnClaim, with its focused stylistic augmentation, can outperform them, especially in scenarios requiring nuanced understanding of specific claim definitions.

Key Results

  • Experiment-I (Individual Datasets): EnClaim consistently outperformed baseline models (BiLSTM, CNN+BILSTM, BERT-base, ClimateBERT) across all three datasets, achieving the highest F1 scores. For Dataset-GCC, EnClaim reached an F1 score of 96%. For Dataset-ECD, it achieved 83%.
  • Experiment-II (Combined Datasets): When all datasets were combined, EnClaim again significantly surpassed BERT, demonstrating the effectiveness of the style-aware approach on a larger, more diverse corpus.
  • Experiment-III (Cross-Dataset Testing): EnClaim showed better performance than BERT when trained on one dataset and tested on another, highlighting its robustness and ability to adapt.
  • Comparison with LLMs: EnClaim outperformed LLAMA-2 13B in terms of Precision, Recall, and F1 scores on the ECD dataset, particularly in few-shot and zero-shot scenarios. This was attributed to EnClaim's adherence to the precise definition of environmental claims.
  • Fine-tuned Mistral-7B: While fine-tuning Mistral-7B on the environmental domain improved its performance, its "hallucinatory property" limited its accurate classification to only 25% of cases, with only 22% of those being correct.

Limitations and Future Work

Limitations:

  • Dataset-SCDC's Multi-Class Nature: The paper notes that the poor performance on Dataset-SCDC in some experiments is partly due to its original multi-class nature, which can be challenging to adapt to binary classification.
  • LLM Hallucinations: The paper highlights the challenge of LLM hallucinations, where models may generate incorrect or nonsensical outputs, as seen with the fine-tuned Mistral-7B.
  • Generalization to Tweets: The paper mentions that GCC, consisting of tweets, might be less representative of full sentences, potentially impacting performance metrics.

Future Work:

The authors propose expanding EnClaim's granularity by categorizing claims into subcategories (e.g., pollution, resource use) and identifying specific environmental aspects. This could be achieved through hierarchical taxonomies, domain-specific knowledge integration, and named entity recognition. Additionally, incorporating multi-label classification and sentiment analysis could provide a richer understanding of claim complexity and impact, ultimately contributing to more informed decision-making and sustainability efforts.

Summary updated

BibTeX
@inproceedings{saha2024enclaim,
  author = {Diya Saha and Manjira Sinha and Tirthankar Dasgupta},
  title = {{EnClaim: A Style Augmented Transformer Architecture for Environmental Claim Detection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure