Transfer Learning for Causal Sentence Detection

Manolis Kyriakakis, Ion Androutsopoulos, Artur Saudabayev, Joan Ginés i AmetlléACL 2019

TL;DR

This paper tackles the challenge of automatically identifying sentences that express causal relationships, a crucial step for mining causal knowledge from text. They investigate how transfer learning with large language models like BERT and ELMO impacts this task, finding that while beneficial for small datasets, its advantage diminishes significantly with larger datasets, suggesting a performance plateau for current neural methods.

What Problem Does This Paper Address?

Understanding cause-and-effect relationships is fundamental to many scientific and medical questions, from disease causes to drug efficacy. While randomized controlled trials are the gold standard, extracting causal insights from vast amounts of text, like scientific literature, is a significant challenge. This is often framed as a "causal sentence detection" task, a precursor to more detailed "relation extraction" that identifies specific entities and their causal roles. However, existing datasets for relation extraction are often sparse in causal instances, making it difficult for models to learn effectively. This paper aims to address this by exploring transfer learning techniques to improve causal sentence detection, particularly in the biomedical domain, and to understand the impact of dataset size on these methods.

Methods

The authors propose a baseline model called BIGRUATT, a bidirectional GRU with self-attention, which processes sentences as sequences of word embeddings. They then explore two transfer learning approaches:

  • BIGRUATT+ELMO: This model augments the BIGRUATT baseline by concatenating word embeddings from ELMO (Embeddings from Language Models), a pre-trained contextual word embedding model.
  • BERT+LR: This approach uses BERT (Bidirectional Encoder Representations from Transformers), a powerful pre-trained transformer model, and adds a simple logistic regression layer on top for classification. The BERT model is fine-tuned on the causal sentence detection task.
  • BERT+BIGRUATT: To investigate if deeper task-specific models improve performance with BERT, they replace the logistic regression layer in BERT+LR with their BIGRUATT model, using BERT's contextual embeddings as input.

For comparison, they also include a simpler LR (n-grams) model using TF-IDF features.

Datasets

The paper utilizes a combination of existing and newly introduced datasets:

  • SemEval-2010 (Task 8): A general relation extraction dataset containing 1,325 causal sentences out of 10,674 total.
  • Causal-TimeBank (CausalTB): A dataset focused on causal sentences, with 244 causal instances.
  • Event StoryLine (EventSL): Another dataset for causal sentence detection, containing 77 causal sentences.
  • BioCausal-Large: A newly developed, large-scale biomedical causal sentence detection dataset with 7,562 causal sentences from PubMed.
  • BioCausal-Small: A publicly available subset of BioCausal-Large, containing 1,113 causal sentences.

To address class imbalance in the first three datasets, they randomly sampled non-causal sentences to create roughly balanced ratios. All datasets were split into training, validation, and test sets.

Key Insights and Ideas

  • Transfer learning's diminishing returns: The most striking finding is that transfer learning (ELMO, BERT) significantly boosts performance only on very small datasets (hundreds of instances). Beyond this, performance plateaus, and larger datasets or more complex models don't yield substantial improvements.
  • Dataset size matters: The paper highlights that for neural methods, there seems to be a point where increasing dataset size or employing advanced transfer learning offers little to no benefit, suggesting current models might be reaching their capacity limits for this task.
  • BERT+BIGRUATT vs. BERT+LR: Combining BERT with a deeper task-specific model (BIGRUATT) generally outperformed a simpler logistic regression layer on BERT, suggesting that shallow task-specific layers might not always be optimal when using powerful pre-trained models.
  • Biomedical vs. Generic: The findings regarding transfer learning's effectiveness are consistent across both generic and biomedical text datasets, indicating a general trend rather than domain-specific behavior.

Key Results

  • On smaller datasets like CausalTB and EventSL, transfer learning methods (BIGRUATT+ELMO, BERT+LR, BERT+BIGRUATT) showed statistically significant improvements in AUC over the baseline BIGRUATT. For instance, on CausalTB, AUC scores ranged from 73.91% (BIGRUATT) to 85.97% (BERT+BIGRUATT).
  • On larger datasets (SemEval, BioCausal-Small, BioCausal-Large), the AUC differences between transfer learning methods and the BIGRUATT baseline were small and not statistically significant in most cases. For example, on BioCausal-Large, AUC scores were generally in the high 90s across most models, with BERT+BIGRUATT achieving 94.70% and BIGRUATT achieving 93.71%.
  • The simpler LR (n-grams) model performed competitively on the EventSL dataset, likely due to its small size and specific characteristics.
  • BERT+BIGRUATT consistently outperformed BERT+LR in AUC on BioCausal-Large, though the statistical significance was limited to this dataset.

Limitations and Future Work

  • Performance Plateau: The authors acknowledge that current neural methods seem to reach a performance plateau with a few thousand training sentences, regardless of dataset size or transfer learning. This suggests a need for new architectural or training paradigms.
  • Dataset Size Impact: The paper's findings on the limited impact of transfer learning on larger datasets contradict some previous work in other NLP tasks, highlighting the importance of considering dataset size when evaluating transfer learning.
  • Future Work: The authors propose using causal sentence detection as a first step in a pipeline, followed by more computationally intensive relation extraction methods to pinpoint entities and their roles. They also plan to explore how to better integrate causal sentence detection with fine-grained relation extraction.

Summary updated

BibTeX
@inproceedings{kyriakakis2019transfer,
  author = {Manolis Kyriakakis and Ion Androutsopoulos and Artur Saudabayev and Joan Ginés i Ametllé},
  title = {{Transfer Learning for Causal Sentence Detection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure