TL;DR
This paper tackles the challenge of automatically verifying corporate promises, a crucial aspect of accountability and trust. The authors propose a BERT-based model enhanced with R-drop regularization to improve robustness and accuracy in classifying promise status, evidence, timeline, and quality. Their approach achieved competitive results, particularly in binary classification tasks, highlighting the effectiveness of R-drop for this domain.
What Problem Does This Paper Address?
In today's business landscape, companies and governments often make promises related to environmental, social, and governance (ESG) initiatives. However, verifying whether these promises are actually fulfilled is a complex and labor-intensive task. This difficulty stems from the diverse nature of industries, languages, and the sheer volume of information involved. The SemEval-2025 Task 6 aims to automate this process by extracting key information from promise statements, including identifying the promise itself, supporting evidence, the clarity of the evidence-promise link, and the appropriate timing for verification. This paper focuses on addressing this challenge for the English dataset within the task.
Methods
The core of the proposed system is a BERT-base model, a powerful pre-trained language model known for its ability to understand context bidirectionally. To enhance its performance and robustness, the authors introduce R-drop (Regularized Dropout). R-drop is a regularization technique that, unlike standard dropout, aims to improve the stability of non-target classes by applying regularization constraints. This encourages the model to produce more consistent predictions across different training cycles, thereby improving its generalizability and reducing overfitting.
The system architecture consists of:
- Input Layer: Processes text data and converts it into BERT's input format (token IDs).
- Context Encoder: Utilizes the pre-trained BERT model to capture deep contextual relationships within the text.
- Dropout Layer: Implements R-drop for regularization.
- Linear Classifier: Maps the BERT-extracted features to the target labels for each subtask, using a fully connected layer and cross-entropy loss.
The loss function is a combination of Cross-Entropy Loss and Kullback-Leibler (KL) Divergence. KL Divergence is used as a regularization term to ensure consistency between model outputs across different training steps.
Datasets
The paper utilizes the Multilingual Dataset for Corporate Promise Verification for SemEval-2025 Task 6. This dataset is designed to evaluate corporate ESG promises and their implementation and includes data in English, French, Chinese, Japanese, and Korean. The authors specifically focus on the English portion of this dataset. The dataset is provided in JSON and Parquet formats, which were converted to TSV for easier processing.
Key Insights and Ideas
- R-drop for Robustness: The key insight is that R-drop regularization significantly improves the robustness and generalizability of BERT models for promise verification tasks, especially in handling noisy or complex data.
- Binary vs. Multi-class Performance: The model demonstrates strong performance in binary classification subtasks (Promise Status and Evidence Status), indicating its effectiveness in simply identifying the presence of a promise or evidence.
- Challenges in Nuance: The model struggles more with multi-class subtasks (Verification Timeline and Evidence Quality), suggesting that capturing nuanced information like specific timelines or the precise clarity of evidence is more challenging for current models.
- Importance of Regularization: The ablation study clearly shows that R-drop contributes positively to the overall performance, underscoring the value of advanced regularization techniques beyond standard dropout.
Key Results
The paper reports competitive results, with the system ranking 9th in the English leaderboard for SemEval-2025 Task 6. Specific accuracies mentioned include:
- Promise Status: 0.7875 (with R-drop) vs. 0.75 (without R-drop)
- Evidence Status: 0.6753 (with R-drop) vs. 0.6125 (without R-drop)
- Verification Timeline: 0.4252 (with R-drop) vs. 0.3756 (without R-drop)
- Evidence Quality: 0.3 (with R-drop) vs. 0.2752 (without R-drop)
The results highlight the effectiveness of the proposed method, particularly in the binary classification tasks.
Limitations and Future Work
The authors acknowledge that while their model performs well on binary classification tasks, its accuracy is lower for the more complex multi-class subtasks of Verification Timeline and Evidence Quality. They attribute this to the difficulty in clearly judging the clarity of evidence and specific deadlines.
Future work suggested includes:
- Text Data Augmentation: Employing techniques like synonym replacement, random insertion, or back-translation to expand the training set.
- Weighted Loss Functions/Oversampling: Addressing potential class imbalance issues by using weighted loss functions or oversampling/undersampling techniques to prevent the model from being biased towards the majority class.
Summary updated