Abstract
This paper introduces a novel framework for Belief-Desire-Intention (BDI) agents to autonomously manage multiple "temporally extended goals" (TEGs)—complex objectives involving long-term constraints and sequences. By representing these goals as Reward Machines (RMs) and using Single-Player Monte-Carlo Tree Search (SP-MCTS) to schedule actions, the authors enable agents to efficiently interleave tasks and mix human-written symbolic plans with reinforcement learning policies. Evaluated in a Mars Rover domain, the system demonstrates high reliability and scalability, achieving a 100% success rate with up to 11 concurrent goals and maintaining high performance as complexity increases.
Introduction
Traditional BDI architectures excel at modeling agent behavior through simple achievement goals (e.g., "reach location A"). However, they often struggle with temporally extended goals that require maintaining invariants over time or following specific sequences of states. Furthermore, managing "intention progression"—deciding which goal to prioritize when multiple objectives coexist—traditionally requires manual, complex scheduling logic. This research addresses these challenges by automating the scheduling of complex, long-term goals without requiring developers to pre-calculate every possible interaction between plans.
Methodology: Reward Machines and MCTS
The framework combines Linear Temporal Logic (LTL) for defining TEGs with Reward Machines for tracking progress. Each goal is mapped to an RM, a finite state machine that emits rewards upon goal completion or penalties (negative infinity) for invariant violations.
To solve the scheduling problem, the agent employs SP-MCTS to simulate future action sequences, selecting paths that maximize total utility across all active RMs. This "neuro-symbolic" approach allows for the seamless integration of traditional symbolic plans with neural network-based reinforcement learning policies. By updating all RMs after every action, the agent can exploit synergies where one action contributes to multiple objectives simultaneously.
Experimental Evaluation
The approach was tested in a customized Mars Rover grid-world environment ($21 \times 21$). The rover must perform soil experiments at various locations while avoiding hazards ("holes") and managing a limited battery capacity of 50 units. The primary constraints involve ensuring the battery never reaches zero and avoiding all hazards while maximizing completed experiments.
Results and Discussion
The system demonstrated exceptional reliability and efficiency. In the Mars Rover domain, the agent achieved a 100% success rate for up to 11 concurrent top-level goals. Even when the number of goals increased to 15—a scenario where battery limits might make a perfect solution impossible—the agent still completed over 96% of its objectives.
The MCTS scheduler successfully learned to prioritize recharging the battery at critical moments while maximizing experiment throughput. Notably, the computational overhead remained low, with the system returning decisions for 15 concurrent intentions in approximately 50 milliseconds on standard hardware.
Conclusion
By bridging the gap between high-level symbolic reasoning and low-level execution through Reward Machines and MCTS, this framework provides a robust method for managing complex, interleaved goals in BDI agents. While the current model assumes deterministic outcomes in a grid-world setting, future work will focus on extending these capabilities to non-deterministic environments and high-fidelity continuous simulators.
Summary updated