TL;DR
AstroMemBench is a new benchmark designed to evaluate how well autonomous AI agents use operational history to navigate the unique constraints of space missions, such as communication delays and safety-critical resource management. The study reveals that while modern memory systems excel at recalling static instructions and facts, they still struggle with the dynamic state tracking and resource planning essential for long-term spacecraft autonomy.
What Problem Does This Paper Address?
As we push toward long-term autonomous operations in planetary exploration and on-orbit servicing, we are increasingly relying on "embodied agents"—AI systems that can perceive, reason, and act within a physical environment. For these agents to succeed, they need a robust memory system. They must remember superseded instructions during communication delays, track the cumulative wear and tear on mechanical parts, and manage dwindling energy reserves over long durations.
However, existing memory benchmarks are largely "conversational" (focused on chat history) or "terrestrial" (focused on simple navigation or object manipulation). They don't capture the high-stakes, resource-constrained, and deterministic nature of space operations. There is a significant gap in understanding how different memory architectures—like Retrieval-Augmented Generation (RAG) or persistent memory modules—actually impact an agent's ability to complete a mission safely and efficiently.
Methods
The authors propose a novel Scenario–Challenge–Capability (SCC) design method to ensure the benchmark is both comprehensive and relevant to the domain.
- Scenario: Nine tasks across three categories: planetary surface operations (e.g., area surveying), on-orbit servicing (e.g., equipment repair), and deep-space exploration (e.g., scientific observation).
- Challenge: 16 specific memory subchallenges derived from ESA’s "Technology 2040" report, covering communication delays, time-criticality, environmental uncertainty, physical interactions, resource constraints, and safety.
- Capability: 10 foundational agent skills, such as spatial navigation, mechanical manipulation, and task planning.
The benchmark uses a deterministic environment to score agents on a scale of 0–100 based on task completion, adherence to safety protocols, and resource usage. They compared several memory configurations: a native baseline (Pi Agent), RAG, and specialized memory systems like Mem0, Chain-of-Memory (CoM), and LightMem.
Datasets
The authors introduced AstroMemBench 1.0, which consists of:
- 140 templates crossing the SCC dimensions.
- 2,240 unique instances (840 easy, 840 medium, 560 hard).
- Long-context history: Each instance includes 30 historical interaction turns, totaling approximately 70,000 tokens in the "background" version, simulating a long-running mission.
Unlike static Q&A datasets, these instances require the agent to ingest history and then act within a simulated environment where past events directly dictate the correct next step.
Key Insights and Ideas
- The "Restart" Resilience: One of the most critical findings is how memory systems handle session restarts. While RAG and native context management perform well in a single continuous session, specialized systems like Mem0 are significantly better at "picking up where they left off" after a system reboot or session break.
- Fact-Centered Bias: Current memory systems have a "retrieval-oriented inductive bias." They are excellent at finding a specific "needle" (like a specific instruction or a past result) in a large "haystack" of data.
- The State-Tracking Struggle: Conversely, agents struggle with "ongoing state maintenance." Remembering a static rule is easy; tracking the cumulative energy consumed or the current configuration of a multi-stage mechanical repair is much harder.
- Instruction Retention vs. Resource Planning: The benchmark showed that memory systems provide the biggest gains in instruction retention (remembering what to do) and result reuse (not recalculating complex parameters), but offer less help for energy reservation and ongoing state tracking.
Key Results
- Performance Gains: In "new-session" evaluations (simulating a restart), Mem0 outperformed RAG by 12.25 points and the native baseline by over 54 points.
- Same-Session Competition: In a single continuous session, the advantage of specialized memory systems over simple RAG was marginal (only 1.39–3.87 points), suggesting that for short-term tasks, native context window management is still highly competitive.
- Efficiency Trade-offs: The authors measured "Score Efficiency" (points earned per billion tokens). While memory systems improve accuracy, they often do so at the cost of significantly higher token usage (up to 46% more than the baseline), which is a critical consideration for power-limited spacecraft.
Limitations and Future Work
The authors acknowledge that the current version of AstroMemBench uses read-only historical turns during the ingestion phase, meaning the agent isn't learning or updating its memory during the 30-turn background history.
Future work aims to:
- Incorporate long-term continuous tasks where the agent must update its memory dynamically over thousands of turns.
- Introduce richer multimodal interactions, moving beyond text and frozen images to more complex sensor data.
- Refine the benchmark to better test procedural learning—the ability of an agent to improve its "how-to" knowledge based on past failures and successes in the mission.
Summary updated