TL;DR
This paper introduces an LLM-based interactive assistant designed to bridge the gap between complex automated trajectory planners and the science teams who use them. By wrapping a Monte Carlo Tree Search (MCTS) planner in a ReAct-based framework with tool access and retrieval-augmented generation, the system achieves 90.9% faithfulness in explaining uncertainty-aware lunar rover paths, a massive 65% improvement over standard zero-shot prompting.
What Problem Does This Paper Address?
Planning a lunar rover mission, like NASA’s VIPER, is a logistical nightmare. Scientists must balance a strict 24-hour time budget, navigation to high-value drill sites, and the physical constraints of a solar-powered vehicle. Crucially, the rover’s speed (Speed Made Good, or SMG) varies based on the sun's position—driving directly toward or away from the low-hanging lunar sun can blind navigation cameras, requiring complex "tacking" maneuvers.
While automated planners like MCTS can quickly find optimal paths through these uncertainties, they are often "black boxes." A science team needs to know why a specific route was chosen and how a 4-hour delay or a new point of interest (POI) might affect mission success. Currently, this verification is done manually, which is too slow for the shifting timelines of active mission planning.
Methods
The authors developed a framework that pairs a rigorous physics-based simulator with a Large Language Model (LLM) assistant.
- The Planner: An MCTS-based solver handles a semi-Markov Decision Process (SMDP). It plans "macro-actions" between waypoints (drill sites, POIs) while an A* algorithm handles the fine-grained navigation between them. The simulator accounts for sun-azimuth-dependent speeds and probabilistic delays (wheel slip, comms shadows).
- The LLM Assistant: Built using the ReAct (Reasoning and Acting) framework, the assistant doesn't just guess answers. It uses a Tool Catalog to interact with the planner. Tools include
Run science station,Change time,Add POI, andRisk assessment(which runs Monte Carlo rollouts to estimate the probability of finishing on time). - Retrieval Layer: To ensure the LLM uses correct domain terminology and mission-specific facts, it uses Retrieval-Augmented Generation (RAG) to search through mission documents and the planner's own source code.
- Guardrails: The LLM is prompted to avoid "AI-speak" (like "reward functions" or "value estimates") and instead use the "Operator Vocabulary" familiar to NASA scientists (e.g., "ice quality," "time budget").
Datasets
The system was evaluated using a custom Query Bank of 115 science-team questions. These were categorized into:
- Plan Comparisons: "How does the plan change if we start 4 hours later?"
- Feasibility: "What is the risk of not reaching the goal?"
- Lookup: Questions about mission docs or planner logic.
- Edge Cases: Ambiguous or impossible requests (e.g., placing a drill site outside the map).
Evaluation was performed using "LLM-as-a-judge" (GPT-4 and Claude Opus), validated against human labels to ensure the judge's scores were trustworthy.
Key Insights and Ideas
- Explainability via Interaction: Instead of putting the LLM inside the planning loop (where it might make unsafe decisions), the authors use it for post-planning explanation. This allows the LLM to act as a translator for a trusted, deterministic planner.
- Grounding is Mandatory: Zero-shot LLMs are terrible at this task (26.7% faithfulness). They tend to hallucinate plausible-sounding but physically impossible lunar physics. Only by giving the LLM specific tools to "ask" the planner for data did faithfulness reach acceptable levels.
- The "Why" remains difficult: While the assistant was excellent at "What" and "How" questions, it still struggled with deep "Why" questions (e.g., "Why did the planner prioritize this site over that one?"). This suggests that even with RAG, some internal logic of MCTS remains hard for LLMs to parse without more explicit symbolic traces.
Key Results
- 90.9% Faithfulness: The full assistant (ReAct + Tools + RAG) was highly accurate in its factual claims compared to the planner's actual output.
- Planner Sensitivity: The underlying MCTS planner correctly responded to physical intuition—for example, shifting the start time by 12 hours completely changed the optimal route to account for new sun angles and shadows.
- Risk Assessment: The
Risk assessmenttool provided crucial data, showing that while a plan might take 20.15 hours deterministically, uncertainty creates a "tail" of outcomes extending toward the 24-hour limit.
Limitations and Future Work
- Hallucinations: Even the best model (Claude Opus) occasionally fabricated details, such as inventing a per-segment breakdown of time slack that the tools never provided.
- Task Success: While faithfulness was high, "Task Success" (calling the exact right tool for a complex query) was only around 50%. The LLM sometimes chose a "good enough" tool rather than the optimal one.
- Future Directions: The authors plan to extend the MDP to handle partial observability (where the rover doesn't know the ice concentration until it arrives) and to test the framework with different types of planning algorithms beyond MCTS.
Summary updated