Abstract
The main purpose of this article is to evaluate the effectiveness of various methods and models for counterfactual determination and the detection of causal knowledge. Counterfactual reasoning is widely used across various fields and holds significant potential in Natural Language Processing (NLP) for improving sentence correctness. For SemEval-2020 Task 5, we pre-processed the official dataset using case conversion, stemming, and abbreviation replacement. Our approach utilizes the last five layers of a Bidirectional Encoder Representations from Transformers (BERT) model along with a Term Frequency-Inverse Document Frequency (TF-IDF) vectorizer for counterfactual detection. Additionally, multi-sample dropout and cross-validation were employed to enhance model generalization and prevent overfitting. Our team, Ferryman, achieved 8th place in Sub-task 1 of the competition.
Introduction
Counterfactual reasoning—imagining alternative realities to learn from the past or predict the future—is a cornerstone of human cognition. In Natural Language Processing (NLP), detecting these statements is notoriously difficult because they often rely on the subjunctive mood and complex causal relationships (e.g., "If the price of oil hadn't dropped, the stock market wouldn't be so low"). The challenge lies in the fact that these statements describe things that did not happen, requiring a model to understand not just the words on the page, but the underlying logic of "what if." Standard models often struggle with the distribution shift between factual and counterfactual data and can easily overfit on specific keywords rather than grasping the structural logic of the sentence.
Methodology
The "Ferryman" team proposed a hybrid architecture that blends modern transformer-based embeddings with classical statistical features. Their pipeline consists of several key stages:
- Preprocessing: Beyond standard lowercasing and stemming, the team implemented abbreviation replacement (e.g., converting "you're" to "you are"). This is particularly important for informal text where grammatical cues for counterfactuals might be obscured by contractions.
- Feature Engineering (The "Last-5 [CLS]" Trick): Instead of relying solely on the final hidden state of the BERT model, the team extracted the
[CLS]token representations from the last five layers of BERT. This captures a richer, hierarchical view of the sentence's semantics. - Hybrid Ensemble: They combined the BERT representations with a TF-IDF (Term Frequency-Inverse Document Frequency) vectorizer. While BERT handles context, TF-IDF ensures the model pays attention to the statistical importance of specific words that might signal counterfactual intent.
- Multi-Sample Dropout: To combat overfitting and accelerate training, they used a multi-sample dropout technique. This involves applying multiple different dropout masks to the same input in a single pass, which helps the model generalize better to unseen data.
Experimental Setup
The study utilized the official dataset provided for SemEval-2020 Task 5 (Detecting Counterfactuals). The primary focus was Sub-task 1, a binary classification task to determine whether a given statement is counterfactual. The dataset includes diverse examples ranging from economic forecasts to medical scenarios, reflecting the multi-disciplinary nature of counterfactual reasoning.
Results and Discussion
The authors conducted an ablation study to show the incremental value of each component. Initial comparisons against baseline models like LightGBM and standard TF-IDF showed that a basic BERT model already provided a significant performance boost.
Further optimization of the BERT model through multi-sample dropout, TF-IDF ensembling, and the inclusion of the last five [CLS] layers led to consistent improvements in the F1 score.
The final ensemble reached an F1 score of 0.856, securing the 8th place in the competition. These results suggest several key insights:
- Context + Statistics: BERT alone is not a silver bullet. Adding a TF-IDF ensemble shows that explicit statistical word importance provides a valuable signal that deep neural networks might occasionally "smooth over."
- Layer Depth: The decision to use the last five layers of BERT suggests that information required to identify counterfactuals—both structural (grammar) and semantic (meaning)—is distributed across the upper layers of the transformer stack.
- Robustness: Multi-sample dropout was crucial for stability, preventing the model from becoming too confident in specific training patterns.
Conclusion
While the Ferryman team's approach demonstrates strong performance in detecting counterfactuals, future work could address the extraction of specific cause-and-effect relationships (Sub-task 2). Additionally, while the model is accurate, the computational overhead of ensembling and multi-layer extraction remains a consideration. Further research into the interpretability of the "Last-5 [CLS]" tokens could reveal which specific linguistic features, such as the subjunctive mood or causal connectors, the model is prioritizing.
Summary updated