Abstract
Humans observe only part of their environment at any moment but can still make complex, long-term decisions thanks to our long-term memory. To test how an AI can learn and utilize its long-term memory, we have developed a partially observable Markov decision processes (POMDP) environment, where the agent has to answer questions while navigating a maze. The environment is completely knowledge graph (KG) based, where the hidden states are dynamic KGs. A KG is both human- and machine-readable, making it easy to see what the agents remember and forget. We train and compare agents with different memory systems, to shed light on how human brains work when it comes to managing its own memory. By repurposing the given learning objective as learning a memory management policy, we were able to capture the most likely hidden state, which is not only interpretable but also reusable.
Introduction
In the quest for artificial general intelligence, one of the most significant hurdles is partial observability. Most real-world environments do not provide a complete picture of the state at any given moment; agents must explore, remember past observations, and synthesize a mental map to make informed decisions. While deep learning has made massive strides, current neural approaches—such as LSTMs and Transformers—often struggle with "long-horizon" tasks. These models store information in "black box" vector embeddings, leading to two major issues: a lack of interpretability and scaling failures. As the memory buffer grows, neural models frequently fail at the "credit assignment" problem, struggling to identify which specific historical data point is relevant to a current query.
This research proposes that because the world is naturally structured into entities, relations, and locations, an agent's memory should reflect this structure. By utilizing Temporal Knowledge Graphs (TKGs), agents can maintain a structured, human-readable, and machine-queryable record of their environment.
Methods and Environment
The researchers introduced Room Environment v3, a benchmark designed to test memory and reasoning. The environment is a $7 \times 7$ grid of 49 rooms, each containing various objects and entities. Unlike previous benchmarks, this environment is fully "graph-shaped," with dynamics that include moving objects and periodic "inner walls" that open and close.
The study compared four distinct agent architectures:
- Symbolic-TKG: This agent uses RDF-star, an extension of the Resource Description Framework. It stores observations as triples (e.g.,
Ball,at_location,Kitchen) but adds "qualifiers" such astime_added,last_accessed, andnum_recalled. - Symbolic-RDF: This agent stores plain triples without temporal metadata, serving as a baseline for the importance of temporal tracking.
- Neural-LSTM & Neural-Transformer: These agents receive the same data but store it as a sequence of tokens in a fixed-length buffer, trained via Deep Q-Networks (DQN).
The TKG agent operates on deterministic rules, using Breadth-First Search (BFS) on its internal map for exploration and SPARQL queries for question answering, prioritizing the most recently or frequently accessed facts.
Key Results and Insights
The experimental results revealed a massive performance gap between symbolic and neural approaches. At a memory capacity of 512 entries, the Symbolic-TKG agent reached 46.52% QA accuracy, while the best neural baseline (LSTM) achieved only 11.2%.
Several key insights emerged from the data:
- Scaling of Structured Memory: In neural models, increasing memory capacity often leads to diminishing returns or performance plateaus. In contrast, the Symbolic-TKG agent's accuracy scaled effectively with memory. Because it uses structured SPARQL queries, adding more data does not make the search process inherently more difficult; it simply makes the internal "map" more complete.
- Exploration Collapse: A striking finding was that neural agents eventually stopped exploring, often getting stuck oscillating between a few familiar rooms. The symbolic agents, driven by BFS on their internal graphs, consistently achieved 100% room coverage by step 70.
- The Necessity of Temporal Qualifiers: Simply remembering a fact is insufficient in a dynamic world. By tracking
last_accessedandtime_added, the TKG agent could handle moving objects and changing environment states far more effectively than the plain RDF agent.
Conclusion
This research makes a compelling case for moving beyond monolithic neural networks toward structured, temporal memory systems. Symbolic Knowledge Graphs provide the interpretability and scaling efficiency required for complex navigation and reasoning tasks. While the current environment uses symbolic observations, future work should focus on "Neuro-Symbolic" hybrids that can translate raw sensory data into structured triples while learning optimal memory management policies. By mirroring how humans organize knowledge, we can build agents that truly "understand" and navigate the world.
Summary updated