Ferryman as SemEval-2020 Task 5: Optimized BERT for Detecting Counterfactuals

Weilong Chen, Yan Zhuang, Peng Wang, Feng Hong, Yan Wang, Yanru Zhang2020 · DOI

Abstract

The main purpose of this article is to evaluate the effectiveness of various methods and models for counterfactual determination and the detection of causal knowledge. Counterfactual reasoning is widely used across various fields and holds significant potential in Natural Language Processing (NLP) for improving sentence correctness. For SemEval-2020 Task 5, we pre-processed the official dataset using case conversion, stemming, and abbreviation replacement. Our approach utilizes the last five layers of a Bidirectional Encoder Representations from Transformers (BERT) model along with a Term Frequency-Inverse Document Frequency (TF-IDF) vectorizer for counterfactual detection. Additionally, multi-sample dropout and cross-validation were employed to enhance model generalization and prevent overfitting. Our team, Ferryman, achieved 8th place in Sub-task 1 of the competition.

Introduction

Counterfactual reasoning—imagining alternative realities to learn from the past or predict the future—is a cornerstone of human cognition. In Natural Language Processing (NLP), detecting these statements is notoriously difficult because they often rely on the subjunctive mood and complex causal relationships (e.g., "If the price of oil hadn't dropped, the stock market wouldn't be so low"). The challenge lies in the fact that these statements describe things that did not happen, requiring a model to understand not just the words on the page, but the underlying logic of "what if." Standard models often struggle with the distribution shift between factual and counterfactual data and can easily overfit on specific keywords rather than grasping the structural logic of the sentence.

Methodology

The "Ferryman" team proposed a hybrid architecture that blends modern transformer-based embeddings with classical statistical features. Their pipeline consists of several key stages:

  1. Preprocessing: Beyond standard lowercasing and stemming, the team implemented abbreviation replacement (e.g., converting "you're" to "you are"). This is particularly important for informal text where grammatical cues for counterfactuals might be obscured by contractions.
  2. Feature Engineering (The "Last-5 [CLS]" Trick): Instead of relying solely on the final hidden state of the BERT model, the team extracted the [CLS] token representations from the last five layers of BERT. This captures a richer, hierarchical view of the sentence's semantics.
  3. Hybrid Ensemble: They combined the BERT representations with a TF-IDF (Term Frequency-Inverse Document Frequency) vectorizer. While BERT handles context, TF-IDF ensures the model pays attention to the statistical importance of specific words that might signal counterfactual intent.
  4. Multi-Sample Dropout: To combat overfitting and accelerate training, they used a multi-sample dropout technique. This involves applying multiple different dropout masks to the same input in a single pass, which helps the model generalize better to unseen data.

Experimental Setup

The study utilized the official dataset provided for SemEval-2020 Task 5 (Detecting Counterfactuals). The primary focus was Sub-task 1, a binary classification task to determine whether a given statement is counterfactual. The dataset includes diverse examples ranging from economic forecasts to medical scenarios, reflecting the multi-disciplinary nature of counterfactual reasoning.

Results and Discussion

The authors conducted an ablation study to show the incremental value of each component. Initial comparisons against baseline models like LightGBM and standard TF-IDF showed that a basic BERT model already provided a significant performance boost.

Comparison of baseline models

Further optimization of the BERT model through multi-sample dropout, TF-IDF ensembling, and the inclusion of the last five [CLS] layers led to consistent improvements in the F1 score.

Ablation study of BERT optimization techniques

The final ensemble reached an F1 score of 0.856, securing the 8th place in the competition. These results suggest several key insights:

  • Context + Statistics: BERT alone is not a silver bullet. Adding a TF-IDF ensemble shows that explicit statistical word importance provides a valuable signal that deep neural networks might occasionally "smooth over."
  • Layer Depth: The decision to use the last five layers of BERT suggests that information required to identify counterfactuals—both structural (grammar) and semantic (meaning)—is distributed across the upper layers of the transformer stack.
  • Robustness: Multi-sample dropout was crucial for stability, preventing the model from becoming too confident in specific training patterns.

Conclusion

While the Ferryman team's approach demonstrates strong performance in detecting counterfactuals, future work could address the extraction of specific cause-and-effect relationships (Sub-task 2). Additionally, while the model is accurate, the computational overhead of ensembling and multi-layer extraction remains a consideration. Further research into the interpretability of the "Last-5 [CLS]" tokens could reveal which specific linguistic features, such as the subjunctive mood or causal connectors, the model is prioritizing.

Summary updated

Abstract

The main purpose of this article is to state the effect of using different methods and models for counterfactual determination and detection of causal knowledge. Nowadays, counterfactual reasoning has been widely used in various fields. In the realm of natural language process(NLP), counterfactual reasoning has huge potential to improve the correctness of a sentence. In the shared Task 5 of detecting counterfactual in SemEval 2020, we pre-process the officially given dataset according to case conversion, extract stem and abbreviation replacement. We use last-5 bidirectional encoder representation from bidirectional encoder representation from transformer (BERT)and term frequency-inverse document frequency (TF-IDF) vectorizer for counterfactual detection. Meanwhile, multi-sample dropout and cross validation are used to improve versatility and prevent problems such as poor generosity caused by overfitting. Finally, our team Ferryman ranked the 8th place in the sub-task 1 of this competition.
BibTeX
@misc{chen2020ferryman,
  author = {Weilong Chen and Yan Zhuang and Peng Wang and Feng Hong and Yan Wang and Yanru Zhang},
  title = {{Ferryman as SemEval-2020 Task 5: Optimized BERT for Detecting Counterfactuals}},
  year = {2020},
  doi = {10.18653/v1/2020.semeval-1.84},
}
Figure