RTHN: A RNN-Transformer Hierarchical Network for Emotion Cause Extraction

Rui Xia, Mengran Zhang, Zixiang DingIJCAI 2019 · DOI

TL;DR

This paper introduces RTHN, a novel hierarchical neural network that leverages both RNNs and Transformers to jointly extract emotion causes from text. By modeling relationships between clauses, RTHN significantly improves upon previous methods that treated clauses independently, achieving state-of-the-art results on a benchmark dataset.

What Problem Does This Paper Address?

Emotion Cause Extraction (ECE) is the task of identifying the reasons behind an expressed emotion in text. While previous approaches have tackled ECE using rule-based methods, traditional machine learning, and deep neural networks, a significant limitation has been their tendency to treat each clause in a document as an isolated classification problem. This overlooks the crucial contextual information and interdependencies between clauses, which are vital for accurately inferring causality. For instance, knowing that one clause is a strong candidate for an emotion cause might decrease the likelihood of other clauses being causes, and vice-versa. This paper aims to address this by proposing a model that can simultaneously consider and classify multiple clauses within a document, capturing both correlation and causality.

Methods

The core of the proposed solution is the RNN-Transformer Hierarchical Network (RTHN), a two-layer architecture:

  • Lower Layer: Word-Level Encoder (RNNs): This layer uses multiple Bidirectional LSTMs (Bi-LSTMs), with each Bi-LSTM dedicated to encoding a single clause. A word-level attention mechanism is then applied to aggregate the word-level hidden states into a clause representation. This captures the semantic meaning within each clause.
  • Upper Layer: Clause-Level Encoder (Transformer): This layer employs a stacked Transformer architecture. The clause representations from the lower layer are fed into the Transformer, which excels at modeling relationships between sequential data. The Transformer's self-attention mechanism allows each clause's representation to be updated by considering all other clauses in the document, effectively capturing correlations.

Beyond the standard Transformer, RTHN introduces two novel enhancements:

  • Relative Position Embedding (RPE): Instead of absolute positional information, RPE encodes the relative distance between a clause and the emotion expression clause. This is based on the intuition that causes are often located near the emotion expression.
  • Global Prediction Embedding (GPE): This component incorporates information about the predicted labels of all clauses in the document. It leverages the observation that emotion causes are relatively rare, and the prediction of one cause influences the probability of others. This helps model the causality between clauses.

Datasets

The paper evaluates RTHN on the benchmark ECE corpus released by Gui et al. (2016a). This corpus is notable for defining ECE as a clause-level classification problem, differing from earlier work that treated it as a word-level sequence labeling task. The authors use a standard 9:1 split for training and testing, reporting results based on 10-fold cross-validation. No new datasets were introduced in this paper.

Key Insights and Ideas

  • Hierarchical Modeling is Crucial: The paper strongly suggests that a hierarchical approach, distinguishing between word-level encoding within clauses and clause-level encoding across the document, is more effective for ECE than flat models.
  • Transformer's Superiority for Clause Relations: The Transformer architecture proves to be a more effective clause-level encoder than RNNs for capturing inter-clause relationships, leading to significant performance gains.
  • Relative Position Matters More Than Absolute: Encoding the relative distance to the emotion expression clause is a more impactful feature for ECE than absolute positional information.
  • Global Context Aids Causality: Incorporating global prediction information (i.e., the likelihood of other clauses being causes) significantly enhances the model's ability to infer causality.
  • Class Imbalance is a Challenge: The ECE task suffers from class imbalance (few emotion causes vs. many non-causes), which can lead models to favor predicting non-causes. Hierarchical models that leverage context can better mitigate this.

Key Results

RTHN achieved state-of-the-art performance on the benchmark ECE corpus, improving the F1 score from 72.69% to 76.77%. This represents a significant leap over 12 compared systems. The paper also shows that:

  • Removing global prediction encoding (RTHN-No-GPE) decreased the F1 score by over 3%.
  • Removing relative position encoding (RTHN-No-RPE) led to a substantial performance drop to 0.4145 F1.
  • Using absolute position embedding instead of relative (RTHN-APE) also resulted in poorer performance (0.5694 F1).
  • Both RNN-RNN and Transformer-Transformer hierarchical models performed worse than RTHN, highlighting the benefit of the RNN-Transformer combination.

Limitations and Future Work

  • Sentence-Level Ignorance: The paper notes that for the benchmark corpus, which typically has short documents with fewer than 20 clauses, they ignore the sentence level between clauses and documents. This simplification might not hold for longer, more complex documents.
  • Transformer Layer Count: The authors mention that using only one layer of Transformer in the word-level encoder might be a reason why the TTHN (Transformer-Transformer) model underperformed.
  • Future Work: The paper implicitly suggests exploring more sophisticated ways to integrate sentence-level information and potentially deeper Transformer architectures for word-level encoding. Further investigation into the optimal combination of RNNs and Transformers for different document structures could also be a future direction.

Summary updated

BibTeX
@inproceedings{xia2019rthn,
  author = {Rui Xia and Mengran Zhang and Zixiang Ding},
  title = {{RTHN: A RNN-Transformer Hierarchical Network for Emotion Cause Extraction}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/734},
}
Figure