WC Team at SemEval-2025 Task 6: PromiseEval: Multinational, Multilingual, Multi-Industry Promise Verification leveraging monolingual and multilingual BERT models

Takumi Nishi, Nicole Miu TakagiACL 2025

TL;DR

This paper tackles the challenge of automatically verifying corporate "promises" made in ESG (Environmental, Social, and Governance) statements across multiple languages. The authors fine-tuned BERT models, both monolingual and multilingual, to identify promises and their supporting evidence, achieving competitive results, particularly for English and Japanese. This work contributes to building more transparent and accountable organizations by enabling automated verification of ESG commitments.

What Problem Does This Paper Address?

Corporate ESG statements are increasingly scrutinized for their commitments to sustainability, social responsibility, and ethical governance. These statements significantly influence public trust and corporate reputation. However, the complexity and lack of standardization in these documents make it difficult to systematically verify if stated promises are actually made and supported by evidence. This is a crucial task for ensuring transparency and holding organizations accountable. The SemEval-2025 Task 6: PromiseEval aims to address this by identifying "promises" and "evidence" within company ESG statements across various languages. The challenge lies in handling diverse linguistic expressions and often limited annotated data, especially for non-English languages.

Methods

The core of the WC team's approach involved fine-tuning pre-trained transformer-based language models, specifically BERT and its derivatives, for the promise identification and evidence linking tasks. They employed a strategy of using both monolingual models tailored to specific languages (English, French, Japanese) and a multilingual model (mBERT) to capture cross-lingual patterns.

For English, they used the base BERT model. For French, they leveraged CamemBERT, a BERT model specifically designed for French texts, known for its strong performance on French NLP tasks. For Japanese, they utilized Tohoku BERT, a well-established BERT model fine-tuned on Japanese data. Finally, they used mBERT for a multilingual model to see how it performed across all languages.

The models were fine-tuned on the provided datasets for Subtasks 1 (ESG promise identification) and Subtask 2 (Evidence identification in support of promise). A key challenge addressed was the class imbalance in the "promise_status" label, where "No" promises were significantly less frequent than "Yes" promises. To mitigate potential model bias, the authors incorporated class weights during training.

Datasets

The training and evaluation data for this research were provided by the PromiseEval2025 organizers. This dataset covers five languages: English, French, Japanese, Chinese, and Korean. The data was extracted from ESG reports of various corporate organizations. For this specific work, the team focused on the English, French, and Japanese datasets for Subtasks 1 and 2. The Japanese dataset, in particular, presented a challenge due to the presence of "NA" values in the "evidence_status" label, which were treated as "No" during preprocessing.

Key Insights and Ideas

  • Monolingual vs. Multilingual BERT: The study explored whether language-specific BERT models or a general multilingual BERT model performed better. While both approaches yielded competitive results, the findings suggest that the optimal strategy might depend on the specific task and language.
  • Class Imbalance Mitigation: The authors explicitly addressed the significant class imbalance in promise identification. Their use of class weights was a practical step to ensure that the models didn't become overly biased towards the majority class, leading to more robust performance.
  • Language-Specific Strengths: The Japanese Tohoku BERT model demonstrated particularly strong performance in Subtask 1, outperforming other models in both accuracy and F1 score, even with a notable imbalance in the "promise_status" label. This highlights the potential benefits of highly specialized monolingual models.
  • Task-Specific Performance: The models generally performed better on promise identification (Subtask 1) than on evidence identification (Subtask 2). This suggests that identifying the mere presence of a promise is a more straightforward task for these models than accurately linking it to specific supporting evidence.

Key Results

Subtask 1 (Promise Identification): * Monolingual Models: Achieved high accuracy and F1 scores, with English at 80.25% accuracy and 0.8672 F1, French at 76.75% accuracy and 0.8558 F1, and Japanese at 94% accuracy and 0.9687 F1. The Japanese model was particularly strong. * Multilingual Model: Produced comparable results to monolingual models, slightly underperforming in English and Japanese but performing better with French data compared to the CamemBERT model.

Subtask 2 (Evidence Identification): * Monolingual Models: Accuracy scores were 71.75% (English), 76.75% (French), and 73.5% (Japanese), with F1 scores of 0.7483, 0.8239, and 0.80, respectively. The English model performed the worst in this subtask. * Multilingual Model: Showed better performance in evidence classification for English and Japanese compared to their monolingual counterparts. However, it performed significantly worse than CamemBERT for French and was the worst-performing metric across all models and tasks for this subtask.

Competition Ranking: The team ranked 10th out of 11 for English, 3rd out of 4 for French, and 2nd out of 3 for Japanese. It's noted that the Kaggle submission evaluated all subtasks, making direct comparison for their specific subtasks challenging.

Limitations and Future Work

The authors acknowledge that despite efforts to address label imbalance, their models might still be biased towards "yes" data. They suggest future work could explore techniques like Synthetic Minority Over-sampling Technique (SMOTE), undersampling the majority class, or implementing alternative loss functions such as Focal Loss to further improve robustness. A more thorough comparison between multilingual and language-specific BERT models is also recommended to definitively determine which methodology is superior overall. The paper also notes that the competition submission was evaluated across all subtasks, making it difficult to precisely assess their performance on Subtasks 1 and 2 relative to other participants.

Summary updated

BibTeX
@inproceedings{nishi2025wc,
  author = {Takumi Nishi and Nicole Miu Takagi},
  title = {{WC Team at SemEval-2025 Task 6: PromiseEval: Multinational, Multilingual, Multi-Industry Promise Verification leveraging monolingual and multilingual BERT models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure