Abstract
Agents have been applied to a wide variety of fields, including power systems and spacecraft. Belief-Desire-Intention (BDI) agents, as one of the most widely used and researched architectures, have the advantage of being able to pursue multiple goals in parallel. The problem of deciding “what to do” next at each of the agent’s deliberation cycle is therefore critical for BDI agents, which is defined as the intention progression problem (IPP). Among all existing approaches to IPP, the majority of approaches have overlooked the significance of runtime historical data, thereby limiting the adaptability and decision-making capabilities of agents. In this paper, we propose to incorporate online learning into the current state-of-the-art intention progression approach $S_{A}$ to overcome the above limitations. This approach not only prevents $S_{A}$ from consuming computational resources on ineffective and inefficient simulations, but also significantly improves the execution efficiency of the agent. Especially when dealing with large-scale problem domains, this improvement significantly enhances the planning capability of the agents. In particular, we have proposed the $SA_{Q}$ and $SA_{L}$ schedulers, both of which can learn how to generate “reasonable” rollouts during the simulation phase of MCTS based on historical simulation data at run time. We compare the performance of our approach with the state-of-the-art $S_{A}$ in a range of scenarios of increasing difficulty. The results demonstrate that our approaches outperform $S_{A}$ , both in terms of the number of goals achieved and the computational overhead required.
Introduction
The Intention Progression Problem (IPP) is a fundamental challenge for Belief-Desire-Intention (BDI) agents. These agents must decide which intention to progress at each deliberation cycle to maximize goal achievement. The current state-of-the-art scheduler, $S_A$, employs Monte-Carlo Tree Search (MCTS) to evaluate potential actions through simulations. However, $S_A$ performs random rollouts, ignoring the valuable historical data generated during these "internal rehearsals." This lack of memory leads to inefficient use of computational resources, as the agent repeatedly explores suboptimal paths.
Methodology: Intelligent Rollouts via Online Learning
To overcome the limitations of random simulations, we introduce two novel schedulers: $SA_Q$ and $SA_L$. These schedulers replace random rollouts with "intelligent" ones guided by online learning.
- $SA_Q$ (Q-learning based): This scheduler utilizes a Q-table to store the values of state-action pairs. During MCTS simulations, the agent follows an $\epsilon$-greedy policy, typically selecting the best-known action from history while maintaining a small probability of exploration.
- $SA_L$ (State-Action Tree based): To improve scalability and efficiency, $SA_L$ uses a hierarchical State-Action tree. This structure tracks visit counts and cumulative rewards more effectively than a flat table and allows for efficient data pruning in dynamic environments.
A key feature of both schedulers is the "reasonable" rollout mechanism. Agents only update their learning models if a simulation's reward exceeds the current average, ensuring that only high-quality strategies are reinforced.
Experimental Setup and Benchmarking
The performance of $SA_Q$ and $SA_L$ was evaluated against the baseline $S_A$ using synthetic Goal-Plan Trees (GPTs) generated by genGPT. GPTs represent the hierarchical relationship between an agent's goals, the plans to achieve them, and the atomic actions required.
Benchmarks included scenarios with a tree depth of 8 and 60 environment variables. Experiments were conducted in both static and dynamic environments, where variables changed according to a Poisson process, to test the agents' adaptability to environmental volatility.
Key Results and Insights
The experimental results highlight significant improvements in both goal achievement and computational efficiency:
- Goal Achievement: In static environments with more than 10 goals, $SA_Q$ achieved approximately 0.7 more goals than $S_A$, while $SA_L$ further improved upon $SA_Q$ by 0.6 goals.
- Efficiency: $SA_L$ demonstrated remarkable efficiency, matching the performance of $S_A$ while requiring 10 times fewer iterations ($\alpha=100$ vs $\alpha=1000$). This translates to a substantial reduction in total computational overhead.
- Dynamic Robustness: In changing environments, $SA_L$ maintained its performance lead, proving that online learning can effectively adapt even when historical data has a limited shelf life.
The core insight is that learning from "imagination"—the agent's own simulations—allows for much faster convergence to high-quality decisions than random exploration.
Conclusion and Future Work
Incorporating online learning into MCTS-based intention progression significantly enhances the capabilities of BDI agents. The $SA_L$ scheduler, in particular, offers a scalable and robust solution for complex, dynamic environments. Future research will investigate applying similar learning techniques to the expansion phase of MCTS and extending the framework to multi-agent environments where agents must reason about the intentions of others.
Summary updated