Abstract
Agents in partially observable environments require persistent memory to integrate observations over time. While KGs (knowledge graphs) provide a natural representation for such evolving state, existing benchmarks rarely expose agents to environments where both the world dynamics and the agent's memory are explicitly graph-shaped. We introduce the Room Environment v3, a configurable environment whose hidden state is an RDF KG and whose observations are RDF triples. The agent may extend these observations into a temporal KG when storing them in long-term memory. The environment is easily adjustable in terms of grid size, number of rooms, inner walls, and moving objects. We define a lightweight temporal KG memory for agents, based on RDF-star-style qualifiers (time_added, last_accessed, num_recalled), and evaluate several symbolic baselines that maintain and query this memory under different capacity constraints. Two neural sequence models (LSTM and Transformer) serve as contrasting baselines without explicit KG structure. Agents train on one layout and are evaluated on a held-out layout with the same dynamics but a different query order, exposing train-test generalization gaps. In this setting, temporal qualifiers lead to more stable performance, and the symbolic TKG (temporal knowledge graph) agent achieves roughly fourfold higher test QA (question-answer) accuracy than the neural baselines under the same environment and query conditions.
Introduction
In partially observable environments—such as a robot navigating a house—agents must integrate observations over time to build a coherent understanding of their surroundings. Traditionally, researchers have relied on neural "black-box" memories like LSTMs or Transformers. However, these models are often opaque, making it difficult to interpret why an agent made a specific decision or what it has actually remembered. Furthermore, while Knowledge Graphs (KGs) are a natural way to represent structured data, most existing reinforcement learning benchmarks do not use them to represent the hidden state of the world. This paper introduces Room Environment v3 to bridge this gap, providing a controlled setting to study how explicit, human-readable symbolic memory compares to neural memory in complex, evolving environments.
Room Environment v3
The Room Environment v3 is a configurable $g \times g$ grid world where every element—including rooms, walls, and moving objects—is defined as RDF triples. The environment is dynamic: inner walls open and close on periodic schedules, and objects move according to deterministic rules. This setup allows for a massive state space while maintaining full interpretability of the world state.
Temporal KG Memory (TKG)
To handle partial observability, the authors propose a Temporal KG Memory (TKG) based on RDF-star. Instead of storing raw observations, the symbolic agent attaches "qualifiers" to facts, such as time_added (when first seen), last_accessed (last time queried), and num_recalled (frequency of use). This structured memory allows for sophisticated reasoning and efficient eviction strategies like Least Recently Used (LRU).
Symbolic agents using SPARQL-style pattern matching and deterministic exploration (e.g., Breadth-First Search) were evaluated against neural agents using Deep Q-Networks with LSTM or Transformer encoders.
Experimental Results
The study revealed a significant performance gap between symbolic and neural approaches. At a memory capacity of 512 entries, the best symbolic agent (RDF-star) achieved a 46.52% QA accuracy, while the best neural agent (LSTM) reached only 11.2%.
A key insight was the "exploration collapse" observed in neural agents. While symbolic agents achieved 100% room coverage by step 74, neural agents often stopped exploring new rooms early in an episode, settling into suboptimal behaviors like oscillating between familiar locations. Additionally, the symbolic agents' performance scaled linearly with memory capacity, whereas neural models failed to meaningfully exploit extra information. The use of temporal qualifiers (RDF-star) also led to much more stable performance across different memory constraints compared to plain RDF triples.
Conclusion and Future Work
The results demonstrate that structured symbolic memory provides a significant advantage in interpretability and performance over standard neural architectures in partially observable tasks. While the current environment uses deterministic dynamics and fixed question formats, future work could involve more complex multi-hop reasoning and stochastic environments. The authors also suggest exploring neuro-symbolic hybrids that combine the high-level exploration capabilities of neural networks with the reliable, structured memory of symbolic KGs.
Summary updated