Introduction
Most modern question-answering (QA) systems excel at retrieving specific factoids. For instance, if asked "Who directed Titanic?", a standard system might return "James Cameron." While accurate, this lacks the natural, conversational flow humans prefer, such as "James Cameron directed the Titanic in 1997." Generating these enriched, natural language answers presents a significant challenge for neural models, primarily due to the diverse ways knowledge is stored and the tendency for models to repeat themselves.
The Challenge of Heterogeneity and Repetition
Knowledge typically resides in multiple formats: structured Knowledge Bases (KBs), unstructured text documents (like Wikipedia), or semi-structured Information Extraction (IE) results. Standard sequence-to-sequence models struggle to integrate these "heterogeneous" sources simultaneously.
Furthermore, Recurrent Neural Network (RNN) decoders often fall into a "repetition trap." When multiple memory sources contain the same fact (e.g., both a KB and a document mention the director), the model may lose track of its internal state and generate redundant phrases like "James Cameron directed Titanic and James Cameron directed it."
The HS-CumuAttn Model
To address these issues, researchers from Peking University proposed HS-CumuAttn (Heterogeneous States with Cumulative Attention). This architecture introduces two key innovations:
1. Decoder with Heterogeneous States
Unlike standard decoders that use a single hidden vector, HS-CumuAttn splits the decoder state into three distinct vectors: $h^{\langle N \rangle}$ (Normal), $h^{\langle K \rangle}$ (Key), and $h^{\langle V \rangle}$ (Value). These vectors are specifically designed to bridge the semantic gap between the neural network and external memory by mapping directly to the "Keys" (e.g., director) and "Values" (e.g., James Cameron) of the knowledge source.
2. Cumulative Attention
The "anti-repetition" mechanism, Cumulative Attention, allows the model to maintain a history of what it has already generated. Before selecting the next word from memory, the model performs self-attention on its previous states. This acts as a "checklist," ensuring the model avoids picking the same memory slot twice and enabling it to enrich answers with additional facts without looping.
Experimental Results
The model was evaluated using the WikiMovies dataset, adapted into two versions: a "Synthetic" version for testing logic and a "Wikipedia" version for real-world noise.
- Reduced Redundancy: On the Synthetic dataset, HS-CumuAttn achieved a redundancy score of 0.0983, significantly outperforming baselines like GenQA (0.1109).
- Superior Coverage: The model correctly identified primary answers ($C_{single}$) at a rate of 98.15% and covered multiple answer parts ($C_{part}$) at 77.28%.
- Naturalness: On the noisy Wikipedia dataset, it achieved a BLEU score of 44.97, indicating high similarity to human-written responses.
Conclusion and Future Work
HS-CumuAttn demonstrates that splitting decoder states and tracking attention history can significantly improve the naturalness and accuracy of QA systems. While the model occasionally struggles with word choice consistency (e.g., confusing "directed" with "written") or complex punctuation, it represents a major step toward conversational AI that can seamlessly navigate structured and unstructured knowledge. Future research will focus on more complex memory structures like Knowledge Graphs and multi-step reasoning.
Summary updated