Natural Answer Generation with Heterogeneous Memory

Yao Fu, Yansong FengNAACL 2018 · OpenReview

Introduction

Most modern question-answering (QA) systems excel at retrieving specific factoids. For instance, if asked "Who directed Titanic?", a standard system might return "James Cameron." While accurate, this lacks the natural, conversational flow humans prefer, such as "James Cameron directed the Titanic in 1997." Generating these enriched, natural language answers presents a significant challenge for neural models, primarily due to the diverse ways knowledge is stored and the tendency for models to repeat themselves.

The Challenge of Heterogeneity and Repetition

Knowledge typically resides in multiple formats: structured Knowledge Bases (KBs), unstructured text documents (like Wikipedia), or semi-structured Information Extraction (IE) results. Standard sequence-to-sequence models struggle to integrate these "heterogeneous" sources simultaneously.

Mapping natural language answers to heterogeneous memory sources (KB and documents).

Furthermore, Recurrent Neural Network (RNN) decoders often fall into a "repetition trap." When multiple memory sources contain the same fact (e.g., both a KB and a document mention the director), the model may lose track of its internal state and generate redundant phrases like "James Cameron directed Titanic and James Cameron directed it."

The HS-CumuAttn Model

To address these issues, researchers from Peking University proposed HS-CumuAttn (Heterogeneous States with Cumulative Attention). This architecture introduces two key innovations:

1. Decoder with Heterogeneous States

Unlike standard decoders that use a single hidden vector, HS-CumuAttn splits the decoder state into three distinct vectors: $h^{\langle N \rangle}$ (Normal), $h^{\langle K \rangle}$ (Key), and $h^{\langle V \rangle}$ (Value). These vectors are specifically designed to bridge the semantic gap between the neural network and external memory by mapping directly to the "Keys" (e.g., director) and "Values" (e.g., James Cameron) of the knowledge source.

The HS-CumuAttn decoder architecture featuring split hidden states and cumulative attention.

2. Cumulative Attention

The "anti-repetition" mechanism, Cumulative Attention, allows the model to maintain a history of what it has already generated. Before selecting the next word from memory, the model performs self-attention on its previous states. This acts as a "checklist," ensuring the model avoids picking the same memory slot twice and enabling it to enrich answers with additional facts without looping.

Experimental Results

The model was evaluated using the WikiMovies dataset, adapted into two versions: a "Synthetic" version for testing logic and a "Wikipedia" version for real-world noise.

  • Reduced Redundancy: On the Synthetic dataset, HS-CumuAttn achieved a redundancy score of 0.0983, significantly outperforming baselines like GenQA (0.1109).
  • Superior Coverage: The model correctly identified primary answers ($C_{single}$) at a rate of 98.15% and covered multiple answer parts ($C_{part}$) at 77.28%.
  • Naturalness: On the noisy Wikipedia dataset, it achieved a BLEU score of 44.97, indicating high similarity to human-written responses.

Conclusion and Future Work

HS-CumuAttn demonstrates that splitting decoder states and tracking attention history can significantly improve the naturalness and accuracy of QA systems. While the model occasionally struggles with word choice consistency (e.g., confusing "directed" with "written") or complex punctuation, it represents a major step toward conversational AI that can seamlessly navigate structured and unstructured knowledge. Future research will focus on more complex memory structures like Knowledge Graphs and multi-step reasoning.

Summary updated

BibTeX
@inproceedings{fu2018natural,
  author = {Yao Fu and Yansong Feng},
  title = {{Natural Answer Generation with Heterogeneous Memory}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=H1Z71m-uZS},
}
Figure