Working Memory-Driven Neural Networks with a Novel Knowledge Enhancement Paradigm for Implicit Discourse Relation Recognition

Fengyu Guo, Ruifang He, Jianwu Dang, Jian WangAAAI 2020 · DOI

Introduction

Implicit Discourse Relation Recognition (IDRR) is a fundamental task in natural language processing that involves identifying logical connections between adjacent text units (arguments) without the aid of explicit connectives like "but" or "because." For instance, in the sequence "The weather was cold. We stayed inside," a human reader intuitively infers a causal relationship. However, for computational models, the absence of explicit markers creates a significant "bottleneck."

Current research in IDRR faces two primary challenges: a lack of reciprocal interaction between arguments during encoding and a lack of external context. Humans resolve these ambiguities by utilizing a "working memory" that integrates immediate perceptions with long-term background knowledge. This paper introduces the Knowledge-Enhanced Attentive Neural Network (KANN), a framework designed to mimic this cognitive process.

Methodology

The KANN architecture is structured around the psychological model of working memory, bifurcating information processing into instant and long-term components.

Instant Memory via Mutual Attention

To capture the nuanced interactions between arguments, KANN employs a Mutual Attention mechanism. Unlike traditional models that encode sentences in isolation, mutual attention constructs a matrix of alignment scores between every word in Argument 1 and Argument 2. This approach accounts for "asymmetric interactions," reflecting how the interpretation of a preceding sentence dynamically shifts the focus when reading the subsequent one.

Long-term Memory via External Knowledge

The model incorporates external semantic knowledge from WordNet to simulate long-term memory. Semantic relationships between word pairs across arguments are retrieved and transformed into "semantic connection embeddings" using TransE, a method for representing knowledge graph relations as vectors.

The architecture of the Knowledge-Enhanced Attentive Neural Network (KANN).

The Knowledge Enhancement Paradigm

The core innovation of KANN lies in its dual-pronged fusion strategy:

  1. Implicit Enhancement: The knowledge matrix is mapped directly into the mutual attention matrix, biasing the model's focus toward word pairs with established real-world semantic links.
  2. Explicit Enhancement: Semantic connection embeddings are concatenated with the final argument representations, providing direct features for the classification layer.

Results and Evaluation

KANN was evaluated on the Penn Discourse TreeBank (PDTB), the standard benchmark for IDRR, focusing on the four top-level relations: Comparison, Contingency, Expansion, and Temporal.

Performance comparison of KANN against baseline models on the PDTB dataset.

As shown in the results table, KANN achieves state-of-the-art (SOTA) performance in several categories. Specifically, it reached F1 scores of 43.92% for Comparison and 73.45% for Expansion. In four-way classification, the model achieved an F1 score of 47.90%, surpassing previous benchmarks such as Lan2017 and Lei2018. Ablation studies confirmed that removing the knowledge component led to a significant performance drop, highlighting the critical role of WordNet integration.

Discussion and Limitations

While KANN demonstrates superior performance in most categories, it faces challenges with Temporal relations (scoring 36.33%). This suggests that temporal logic may require deeper reasoning involving verb tenses and world-event sequences that simple word-pair semantics cannot fully capture. Furthermore, the model's reliance on WordNet means its "long-term memory" is limited by the coverage of the database; niche jargon or modern slang may not be effectively processed.

Future work will explore more sophisticated knowledge integration techniques and move toward "deeper inference" mechanisms to better handle complex discourse structures.

Summary updated

BibTeX
@inproceedings{guo2020working,
  author = {Fengyu Guo and Ruifang He and Jianwu Dang and Jian Wang},
  title = {{Working Memory-Driven Neural Networks with a Novel Knowledge Enhancement Paradigm for Implicit Discourse Relation Recognition}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i05.6287},
}
Figure