AstroMemBench: A Benchmark for Memory Systems in Embodied Spacecraft Agents

You Wu, Guhong Zhang, Hongwei Li, Xinyue Hong, Yuan JiaarXiv 2026 · arXiv

TL;DR

AstroMemBench is a new benchmark designed to evaluate how well autonomous AI agents use operational history to navigate the unique constraints of space missions, such as communication delays and safety-critical resource management. The study reveals that while modern memory systems excel at recalling static instructions and facts, they still struggle with the dynamic state tracking and resource planning essential for long-term spacecraft autonomy.

What Problem Does This Paper Address?

As we push toward long-term autonomous operations in planetary exploration and on-orbit servicing, we are increasingly relying on "embodied agents"—AI systems that can perceive, reason, and act within a physical environment. For these agents to succeed, they need a robust memory system. They must remember superseded instructions during communication delays, track the cumulative wear and tear on mechanical parts, and manage dwindling energy reserves over long durations.

However, existing memory benchmarks are largely "conversational" (focused on chat history) or "terrestrial" (focused on simple navigation or object manipulation). They don't capture the high-stakes, resource-constrained, and deterministic nature of space operations. There is a significant gap in understanding how different memory architectures—like Retrieval-Augmented Generation (RAG) or persistent memory modules—actually impact an agent's ability to complete a mission safely and efficiently.

Methods

The authors propose a novel Scenario–Challenge–Capability (SCC) design method to ensure the benchmark is both comprehensive and relevant to the domain.

  • Scenario: Nine tasks across three categories: planetary surface operations (e.g., area surveying), on-orbit servicing (e.g., equipment repair), and deep-space exploration (e.g., scientific observation).
  • Challenge: 16 specific memory subchallenges derived from ESA’s "Technology 2040" report, covering communication delays, time-criticality, environmental uncertainty, physical interactions, resource constraints, and safety.
  • Capability: 10 foundational agent skills, such as spatial navigation, mechanical manipulation, and task planning.
Figure 2: Coverage under the three-dimensional SCC design method. The base crosses nine scenario tasks with 16 subchallenges. Each applicable cell defines a template testing that subchallenge in that scenario task: 140 templates cover all applicable combinations; four inapplicable combinations are hatched. The ten capability layers share these coordinates, with colored cells marking capabilities used by each template (285 associations). Two staggered columns expose every layer’s projection. Each capability spans multiple tasks and subchallenges.

The benchmark uses a deterministic environment to score agents on a scale of 0–100 based on task completion, adherence to safety protocols, and resource usage. They compared several memory configurations: a native baseline (Pi Agent), RAG, and specialized memory systems like Mem0, Chain-of-Memory (CoM), and LightMem.

Datasets

The authors introduced AstroMemBench 1.0, which consists of:

  • 140 templates crossing the SCC dimensions.
  • 2,240 unique instances (840 easy, 840 medium, 560 hard).
  • Long-context history: Each instance includes 30 historical interaction turns, totaling approximately 70,000 tokens in the "background" version, simulating a long-running mission.

Unlike static Q&A datasets, these instances require the agent to ingest history and then act within a simulated environment where past events directly dictate the correct next step.

Key Insights and Ideas

  • The "Restart" Resilience: One of the most critical findings is how memory systems handle session restarts. While RAG and native context management perform well in a single continuous session, specialized systems like Mem0 are significantly better at "picking up where they left off" after a system reboot or session break.
  • Fact-Centered Bias: Current memory systems have a "retrieval-oriented inductive bias." They are excellent at finding a specific "needle" (like a specific instruction or a past result) in a large "haystack" of data.
  • The State-Tracking Struggle: Conversely, agents struggle with "ongoing state maintenance." Remembering a static rule is easy; tracking the cumulative energy consumed or the current configuration of a multi-stage mechanical repair is much harder.
  • Instruction Retention vs. Resource Planning: The benchmark showed that memory systems provide the biggest gains in instruction retention (remembering what to do) and result reuse (not recalculating complex parameters), but offer less help for energy reservation and ongoing state tracking.
Table 2: Typical test construction for the 16 memory subchallenges

Key Results

  • Performance Gains: In "new-session" evaluations (simulating a restart), Mem0 outperformed RAG by 12.25 points and the native baseline by over 54 points.
  • Same-Session Competition: In a single continuous session, the advantage of specialized memory systems over simple RAG was marginal (only 1.39–3.87 points), suggesting that for short-term tasks, native context window management is still highly competitive.
  • Efficiency Trade-offs: The authors measured "Score Efficiency" (points earned per billion tokens). While memory systems improve accuracy, they often do so at the cost of significantly higher token usage (up to 46% more than the baseline), which is a critical consideration for power-limited spacecraft.

Limitations and Future Work

The authors acknowledge that the current version of AstroMemBench uses read-only historical turns during the ingestion phase, meaning the agent isn't learning or updating its memory during the 30-turn background history.

Future work aims to:

  • Incorporate long-term continuous tasks where the agent must update its memory dynamically over thousands of turns.
  • Introduce richer multimodal interactions, moving beyond text and frozen images to more complex sensor data.
  • Refine the benchmark to better test procedural learning—the ability of an agent to improve its "how-to" knowledge based on past failures and successes in the mission.

Summary updated

Abstract

Long-term spacecraft autonomy requires embodied agents to use operational history under communication delays, time-critical operations, environmental uncertainty, complex physical interactions, power and computational constraints, and safety and reliability requirements. We introduce AstroMemBench and a Scenario-Challenge-Capability design method that maps these requirements to executable memory tests. The benchmark covers 16 memory subchallenges through 140 templates and 2240 instances across nine tasks in planetary surface operations, on-orbit servicing, and deep-space exploration. A deterministic environment scores actions for task completion, state changes, safety, and resource use. Ablations under nine conditions show that the benchmark is sensitive to how accurately agents recover and use historical information and to differences in memory-system design. Main experiments compare six configurations. Mem0, Chain-of-Memory, LightMem, and retrieval-augmented generation (RAG) improve same-session mean scores over native Pi Agent, with no clear advantage of the three memory systems over RAG. After a session restart, Mem0 outperforms RAG by 12.25 points. Benefits favor instruction retention and result reuse over ongoing state tracking and energy reservation. AstroMemBench supports the design, development, and selection of memory systems for embodied spacecraft agents. AstroMemBench 1.0 and its evaluation code are publicly available: https://github.com/flyingView/AstroMemBench-v1.
BibTeX
@misc{wu2026astromembench,
  author = {You Wu and Guhong Zhang and Hongwei Li and Xinyue Hong and Yuan Jia},
  title = {{AstroMemBench: A Benchmark for Memory Systems in Embodied Spacecraft Agents}},
  year = {2026},
  eprint = {2611.16590},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16590},
}
Figure