Abstract
Identifying causal relationships in natural language is a fundamental challenge in NLP. This paper describes the system developed by team LTRC for the CASE 2022 shared task on the Causal News Corpus. We employ parameter-efficient adapters integrated into pre-trained Transformer models to perform causal event classification and span detection. Our approach demonstrates that freezing model weights and training small bottleneck layers can yield competitive results, ranking 3rd in causal classification and 4th in span extraction.
Introduction
Event Causality Identification (ECI) involves detecting whether a sentence describes a cause-and-effect relationship and identifying the specific text spans representing the cause, the effect, and the linguistic signal connecting them. This task is critical for applications ranging from financial analysis to socio-political event monitoring. However, full fine-tuning of large language models for such niche tasks is often computationally prohibitive and prone to overfitting.
Methodology
We utilized a Transformer-based architecture enhanced with Adapters. Adapters are small, trainable bottleneck layers inserted between the existing layers of a frozen pre-trained model. This allows for task-specific tuning without modifying the original model weights, preserving general linguistic knowledge.
Subtask 1: Causal Event Classification
For determining if a sentence contains a causal relation, we experimented with BART-large, RoBERTa-large, and RoBERTa-base. We found that RoBERTa-base with an adapter and a simple classification head provided the best balance of performance and efficiency.
Subtask 2: Cause-Effect-Signal Span Detection
This subtask was framed as a sequence labeling problem using IOB (Inside-Outside-Beginning) tagging. We employed BERT-base with adapters to label each token. To ensure valid sequence predictions, we implemented four heuristic post-processing rules to clean the predicted spans.
Data
The primary dataset was the Causal News Corpus (CNC), containing 3,559 event sentences from news articles. To improve model robustness, we augmented the training data with UniCausal, a larger unified benchmark for causal text mining, resulting in a combined training set of 6,767 samples.
Results and Discussion
Our system achieved a binary F1 score of 0.853 in Subtask 1, placing 3rd in the competition. In Subtask 2, we reached a macro F1 score of 0.032, ranking 4th.
The relatively low scores in span detection highlight the extreme difficulty of identifying "signals"—short, common words like "because" or "led to"—which models often overlook. Error analysis further revealed that linguistic ambiguities, such as polysemous words (e.g., "fast" as a noun vs. adjective) and idioms (e.g., "raining bombs"), remain significant hurdles for current Transformer architectures.
Conclusion
This work demonstrates the effectiveness of parameter-efficient adapters for specialized NLP tasks like causality extraction. Future improvements could involve ensemble modeling, the addition of CRF layers to better capture sequence dependencies, and the incorporation of syntactic features like POS tags to disambiguate complex sentence structures.
Summary updated