Abstract
Seungwhan Moon, Pararth Shah, Rajen Subba, Anuj Kumar. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations.
2019.
Introduction
As digital footprints expand, users accumulate vast collections of photos, videos, and social media posts. However, traditional retrieval methods—primarily keyword-based search or chronological timelines—fail to capture the associative nature of human memory. We remember events through their connections to people, locations, and activities. This paper presents a conversational assistant designed for "Memory Grounded" reasoning, enabling users to navigate their personal digital histories through natural, multi-turn dialogue.
System Architecture: The Memory Graph Network
The core innovation of this system is the Memory Graph Network (MGN). Rather than treating memories as isolated entries, the system organizes episodic data into a structured graph where nodes represent events and edges represent shared entities (e.g., participants, locations).
The MGN architecture employs several key components:
- Graph Traversal: The model learns to "walk" the graph to resolve multi-hop queries.
- Neural Reasoning Modules: Specialized sub-networks handle logical operations such as
COUNT(e.g., "How many times did I go to Boston?") orCHOOSE. - Query Encoding: Bi-LSTM encoders process natural language input, while graph embeddings (e.g., TransE) capture the structural context of the memory nodes.
Grounding and Proactive Interaction
A key feature of the assistant is its ability to ground conversation in visual media. The system includes a Photo Recommender module that utilizes attention scores from the MGN to surface relevant photos associated with the current dialogue context.
Beyond simple retrieval, the system mimics human reminiscing by providing proactive suggestions. By identifying "nearby" nodes in the graph that share attributes with the current topic, the assistant can suggest related memories (e.g., "You also visited this park with Jon last summer"), making the interaction feel more like a natural conversation than a database query.
Dataset and Evaluation
To train and evaluate the system, the authors developed a synthetic memory graph generator and a Memory Dialog Dataset. The latter was collected using a Wizard-of-Oz setup where human participants engaged in dialogues grounded in structured memory data.
The system demonstrates significant capabilities in:
- Multi-hop Reasoning: Answering complex questions that require traversing multiple nodes.
- Contextual Awareness: Resolving anaphora and maintaining state across multiple turns by tracking "activated" nodes in the graph.
Limitations and Future Directions
While the proof-of-concept is successful, several challenges remain for real-world deployment. These include the "cold start" problem of extracting high-quality graphs from unstructured user data, ensuring robust privacy for personal information, and scaling graph search as a user's memory history grows over decades.
Summary updated