A Recurrent Neural Model with Attention for the Recognition of Chinese Implicit Discourse Relations

Samuel Rönnqvist, Niko Schenk, Christian ChiarcosACL 2017 · DOI

TL;DR

This paper introduces an attention-based Bi-LSTM model for recognizing implicit discourse relations in Chinese text, outperforming previous word-order-agnostic methods. By treating argument pairs as a joint sequence and incorporating a novel partial sampling scheme, the model achieves state-of-the-art results on the Chinese Discourse Treebank, offering a more interpretable approach to understanding text coherence.

What Problem Does This Paper Address?

Understanding how sentences and phrases connect to form a coherent text is crucial for natural language processing (NLP) tasks like question answering and summarization. Discourse relations, which describe these connections, can be explicit (marked by words like "because") or implicit (where the relationship is implied). Implicit discourse relations are significantly more challenging to detect, especially in Chinese, where they constitute a much larger proportion of relations compared to English. Existing methods often rely on complex, feature-rich approaches or ignore word order, limiting their effectiveness. This paper aims to address the difficulty of implicit discourse relation recognition in Chinese by proposing a novel neural network architecture that effectively models argument sequences and their relationships.

Methods

The core of the proposed method is an attention-based Bidirectional Long Short-Term Memory (Bi-LSTM) network. Here's a breakdown of the key techniques:

  • Bi-LSTM: This recurrent neural network architecture processes the input sequence in both forward and backward directions, allowing it to capture dependencies from both preceding and succeeding tokens. This is crucial for understanding the context of discourse relations.
  • Attention Mechanism: The model incorporates an attention mechanism that learns to dynamically assign weights to different parts of the input sequence. This allows the model to focus on the most informative tokens when making a classification decision, providing interpretability by highlighting which parts of the arguments are most relevant.
  • Joint Sequence Modeling: Unlike some prior work that treats arguments separately, this model treats the two arguments (Arg1 and Arg2) of a discourse relation as a single, joint sequence. This is hypothesized to better capture the interaction between the arguments.
  • Partial Argument Sampling: To augment the training data and improve robustness, the authors introduce a novel sampling scheme. Training instances are expanded to include not only pairs of arguments but also individual arguments. This is motivated by linguistic findings suggesting single arguments can evoke strong expectations about discourse relations and by the machine learning benefit of learning better representations of individual argument spans.
  • Word Embeddings: The model uses pre-trained 300-dimensional Chinese Gigaword word vectors, which are further fine-tuned during training. Special markers are inserted to denote the start and end of argument spans.

Datasets

  • Chinese Discourse Treebank (CDTB): This is the primary dataset used for training and evaluating the model. The paper notes that implicit relations are significantly more prevalent in the CDTB than explicit ones.
  • Penn Discourse Treebank (PDTB): As a side experiment to test language independence, the model was also evaluated on the English PDTB.

Key Insights and Ideas

  • Joint Sequence Modeling is Key: Treating argument pairs as a single sequence, rather than processing them independently, significantly improves performance for implicit discourse relation recognition. This suggests that the interaction and alignment between arguments are critical.
  • Attention for Interpretability: The attention mechanism not only aids in classification but also provides valuable insights into why the model makes certain decisions. Visualizing attention weights can reveal how the model connects semantically related terms across argument boundaries.
  • Partial Argument Sampling Boosts Performance: The novel partial argument sampling scheme, which includes individual arguments in training, is shown to be crucial for achieving competitive results, demonstrating its effectiveness in improving model robustness and generalization.
  • Chinese Implicit Relations are Challenging: The paper highlights the higher prevalence of implicit relations in Chinese and the difficulty in their detection, underscoring the need for specialized models like the one proposed.

Key Results

  • State-of-the-Art on CDTB: The proposed attention-based Bi-LSTM model achieved an accuracy of 73.01% on the CDTB test set, outperforming the best feedforward system (Wang and Lan, 2016) and other word-order-agnostic approaches.
  • Ablation Study Findings:
    • Turning off the attention mechanism resulted in a 2.70% absolute decrease in test set accuracy.
  • Excluding the partial sampling scheme led to a 5.74% absolute decrease in accuracy.
  • Language Independence: On the English PDTB, the model achieved 27.09% accuracy, surpassing the majority class baseline (22.01%), suggesting potential for generalization across languages despite not being optimized for English.

Limitations and Future Work

  • Computational Cost of Attention: While beneficial, the attention mechanism can make the model slightly more difficult to optimize.
  • Future Work: The authors intend to extend their approach to other languages and domains, including narrative story understanding and question answering. They believe recurrent modeling of implicit discourse information can be a significant driver in complex semantic processing tasks.

Summary updated

BibTeX
@inproceedings{ronnqvist2017recurrent,
  author = {Samuel Rönnqvist and Niko Schenk and Christian Chiarcos},
  title = {{A Recurrent Neural Model with Attention for the Recognition of Chinese Implicit Discourse Relations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2017},
  doi = {10.18653/v1/p17-2040},
}
Figure