Temporal Knowledge-Graph Memory in a Partially Observable Environment

Taewoon Kim, Vincent François-Lavet, Michael CochezarXiv 2024 · DOI · arXiv

Abstract

Agents in partially observable environments require persistent memory to integrate observations over time. While KGs (knowledge graphs) provide a natural representation for such evolving state, existing benchmarks rarely expose agents to environments where both the world dynamics and the agent's memory are explicitly graph-shaped. We introduce the Room Environment v3, a configurable environment whose hidden state is an RDF KG and whose observations are RDF triples. The agent may extend these observations into a temporal KG when storing them in long-term memory. The environment is easily adjustable in terms of grid size, number of rooms, inner walls, and moving objects. We define a lightweight temporal KG memory for agents, based on RDF-star-style qualifiers (time_added, last_accessed, num_recalled), and evaluate several symbolic baselines that maintain and query this memory under different capacity constraints. Two neural sequence models (LSTM and Transformer) serve as contrasting baselines without explicit KG structure. Agents train on one layout and are evaluated on a held-out layout with the same dynamics but a different query order, exposing train-test generalization gaps. In this setting, temporal qualifiers lead to more stable performance, and the symbolic TKG (temporal knowledge graph) agent achieves roughly fourfold higher test QA (question-answer) accuracy than the neural baselines under the same environment and query conditions.

Introduction

In partially observable environments—such as a robot navigating a house—agents must integrate observations over time to build a coherent understanding of their surroundings. Traditionally, researchers have relied on neural "black-box" memories like LSTMs or Transformers. However, these models are often opaque, making it difficult to interpret why an agent made a specific decision or what it has actually remembered. Furthermore, while Knowledge Graphs (KGs) are a natural way to represent structured data, most existing reinforcement learning benchmarks do not use them to represent the hidden state of the world. This paper introduces Room Environment v3 to bridge this gap, providing a controlled setting to study how explicit, human-readable symbolic memory compares to neural memory in complex, evolving environments.

Room Environment v3

The Room Environment v3 is a configurable $g \times g$ grid world where every element—including rooms, walls, and moving objects—is defined as RDF triples. The environment is dynamic: inner walls open and close on periodic schedules, and objects move according to deterministic rules. This setup allows for a massive state space while maintaining full interpretability of the world state.

The Room Environment v3 layout showing rooms, objects, and dynamic walls.

Temporal KG Memory (TKG)

To handle partial observability, the authors propose a Temporal KG Memory (TKG) based on RDF-star. Instead of storing raw observations, the symbolic agent attaches "qualifiers" to facts, such as time_added (when first seen), last_accessed (last time queried), and num_recalled (frequency of use). This structured memory allows for sophisticated reasoning and efficient eviction strategies like Least Recently Used (LRU).

A visualization of the Knowledge Graph representing the environment's state and agent observations.

Symbolic agents using SPARQL-style pattern matching and deterministic exploration (e.g., Breadth-First Search) were evaluated against neural agents using Deep Q-Networks with LSTM or Transformer encoders.

Experimental Results

The study revealed a significant performance gap between symbolic and neural approaches. At a memory capacity of 512 entries, the best symbolic agent (RDF-star) achieved a 46.52% QA accuracy, while the best neural agent (LSTM) reached only 11.2%.

A key insight was the "exploration collapse" observed in neural agents. While symbolic agents achieved 100% room coverage by step 74, neural agents often stopped exploring new rooms early in an episode, settling into suboptimal behaviors like oscillating between familiar locations. Additionally, the symbolic agents' performance scaled linearly with memory capacity, whereas neural models failed to meaningfully exploit extra information. The use of temporal qualifiers (RDF-star) also led to much more stable performance across different memory constraints compared to plain RDF triples.

Conclusion and Future Work

The results demonstrate that structured symbolic memory provides a significant advantage in interpretability and performance over standard neural architectures in partially observable tasks. While the current environment uses deterministic dynamics and fixed question formats, future work could involve more complex multi-hop reasoning and stochastic environments. The authors also suggest exploring neuro-symbolic hybrids that combine the high-level exploration capabilities of neural networks with the reliable, structured memory of symbolic KGs.

Summary updated

Abstract

Agents in partially observable environments require persistent memory to integrate observations over time. While KGs (knowledge graphs) provide a natural representation for such evolving state, existing benchmarks rarely expose agents to environments where both the world dynamics and the agent's memory are explicitly graph-shaped. We introduce the Room Environment v3, a configurable environment whose hidden state is an RDF KG and whose observations are RDF triples. The agent may extend these observations into a temporal KG when storing them in long-term memory. The environment is easily adjustable in terms of grid size, number of rooms, inner walls, and moving objects. We define a lightweight temporal KG memory for agents, based on RDF-star-style qualifiers (time_added, last_accessed, num_recalled), and evaluate several symbolic baselines that maintain and query this memory under different capacity constraints. Two neural sequence models (LSTM and Transformer) serve as contrasting baselines without explicit KG structure. Agents train on one layout and are evaluated on a held-out layout with the same dynamics but a different query order, exposing train-test generalization gaps. In this setting, temporal qualifiers lead to more stable performance, and the symbolic TKG (temporal knowledge graph) agent achieves roughly fourfold higher test QA (question-answer) accuracy than the neural baselines under the same environment and query conditions. The environment, agent implementations, and experimental scripts are released for reproducible research at https://github.com/humemai/agent-room-env-v3 and https://github.com/humemai/room-env.
BibTeX
@misc{kim2024temporal,
  author = {Taewoon Kim and Vincent François-Lavet and Michael Cochez},
  title = {{Temporal Knowledge-Graph Memory in a Partially Observable Environment}},
  year = {2024},
  eprint = {2408.05861},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.05861},
}
Figure