TL;DR
This paper tackles the challenge of incorporating external knowledge into non-task-oriented dialogue systems, which often struggle with long-term memory and coherence. They propose a novel, space-efficient method to encode knowledge graphs by restricting attention weights in Transformer models, allowing for richer context without sacrificing performance. This approach leads to more comprehensive and engaging dialogues.
What Problem Does This Paper Address?
Modern neural conversational models, like those based on Transformers (e.g., GPT-2), excel at generating human-like text. However, they often lack "long-term memory" to maintain consistent world knowledge or personality across extended conversations. This can lead to incoherent or factually incorrect responses. To address this, researchers have explored "knowledge-grounded dialogue generation," where models are provided with external knowledge, often in the form of knowledge graphs.
The main hurdle here is the limited input sequence length of Transformer models (e.g., 1024 tokens for GPT-2). Simply adding paraphrased knowledge triples as text into the dialogue history consumes valuable token space. This paper aims to find a more efficient way to represent this structured knowledge, allowing more of it to be included without exceeding the model's capacity, thereby improving dialogue quality.
Methods
The core innovation is a "space-efficient context encoding" method for knowledge graphs. Instead of paraphrasing knowledge triples into natural language sentences, they represent the graph structure directly within the Transformer's attention mechanism. Specifically, they restrict the self-attention layers to only attend to connected nodes within the graph. This is inspired by graph neural networks and message-passing algorithms.
They propose two encoding strategies:
- Series Encoding: Entity and relation tokens are sequenced and added to the word layer, with special tokens marking them as entities or relations.
- Parallel Encoding: Entity tokens are added to the word layer, and relation tokens are added to the segments layer, allowing for a more direct parallel representation.
To further preserve the graph structure, they introduce a per-graph attention mask to all hidden layers. This mask guides the attention mechanism, ensuring that tokens can only attend to their connected neighbors in the graph.
Datasets
The authors used two publicly available human-human multi-turn dialogue datasets enriched with structured knowledge:
- KOMODIS: A closed-domain dataset focused on movie discussions, providing facts, trivia, and opinions about films.
- OpenDialKG: An open-domain dataset collected via a Wizard-of-Oz setup, where dialogues are paired with "KG paths" from Freebase.
Key Insights and Ideas
- Structured Knowledge is Better: Representing knowledge graphs directly, rather than paraphrasing them into text, offers a more concise and efficient way to inject information into Transformer models.
- Attention Masking is Key: Restricting self-attention based on graph structure is crucial for preserving the relationships between entities and preventing information loss.
- Perplexity Isn't Everything: The study highlights that perplexity, a common metric for language models, doesn't always correlate with human judgments of naturalness or overall dialogue quality, especially across different datasets.
- Human Evaluation is Essential: For nuanced tasks like knowledge-grounded dialogue, human evaluation is indispensable for assessing aspects like knowledge precision, consistency, naturalness, and interestingness.
Key Results
- Space Reduction: Their proposed encoding methods reduced the required context space by a factor of up to 3.6 compared to baseline methods (base-triples and base-paraphrased).
- Performance Maintained: Despite the space savings, the models achieved comparable or better results in knowledge precision, consistency, and naturalness compared to baselines.
- More Engaging Dialogues: Human evaluations indicated that dialogues generated with their method were judged as more comprehensive and interesting.
- Graph Depth Impact: For KOMODIS, increasing graph depth (more context) generally improved metrics, though the gains from depth 2 over depth 1 were not always statistically significant. For OpenDialKG, the impact of graph depth was less pronounced, possibly due to the dataset's nature.
- Graph Attention Masking: An early evaluation suggested that graph attention masking significantly improved dialogue quality, particularly for larger subgraphs, validating their hypothesis.
Limitations and Future Work
- Limited Resources: Due to resource constraints, they could not perform a full human evaluation of all experimental variations, specifically limiting the investigation into graph attention masking.
- Dataset Specifics: The performance of graph depth variations differed between KOMODIS and OpenDialKG, suggesting that dataset characteristics (e.g., dialogue structure, entity interchangeability) play a significant role.
- Future Work: The authors suggest further exploration of graph attention masking and more extensive human evaluations to fully understand its impact. They also note that the parallel encoding performed slightly worse than series encoding, potentially due to insufficient training data for the more complex type distinctions in OpenDialKG, suggesting that more data or advanced techniques might be needed for such scenarios.
Summary updated