Abstract
Modular Reinforcement Learning (MRL) allows for the decomposition of complex tasks into manageable sub-problems. However, traditional MRL often relies on additive reward structures, which require manual tuning of reward scales across modules to ensure effective behavior. This paper introduces Arbi-Q, a command arbitration algorithm that enables the composition of RL modules with disparate reward scales. By employing a high-level Command Arbitrator that learns to select the most appropriate module for a given state, Arbi-Q facilitates "plug-and-play" AI development. Experiments in the Bunny-Wolf World demonstrate that Arbi-Q maintains optimal performance even when module rewards are intentionally mismatched, offering a significant step toward reusable RL libraries.
Introduction
In Reinforcement Learning (RL), agents typically learn to maximize a scalar reward signal through trial and error. To address complex real-world tasks, researchers frequently turn to Modular RL (MRL), which breaks a global objective (e.g., "survive in the wild") into specialized sub-tasks such as "find food" and "avoid predators."
Despite its conceptual appeal, MRL faces a significant engineering hurdle: reward scale alignment. Most current MRL systems use additive rewards, simply summing the preferences of all modules to select an action. If one module operates on a scale of 0 to 1 and another on 0 to 100, the agent will prioritize the latter, potentially ignoring critical sub-tasks. This lack of "composability" forces developers to manually tune reward scales, preventing the reuse of RL modules across different projects or by different teams.
The Arbi-Q Framework
To solve the composability problem, we propose Arbi-Q. Instead of a fixed mathematical summation, Arbi-Q introduces a Command Arbitrator that is itself a reinforcement learner. Drawing on social choice theory (Arrow’s Paradox), the system appoints the arbitrator as a "benevolent dictator" that learns through experience which module to trust in specific situations.
Key Components
- Modules as Actions: The arbitrator treats individual modules as its available actions. In any given state, it learns which module's policy is most likely to yield a high global reward.
- Decoupled Rewards: The arbitrator maintains its own high-level reward function (e.g., "overall survival"), which is entirely independent of the internal reward scales used by the sub-modules.
- Algorithm Agnosticism: Arbi-Q can integrate modules regardless of their underlying learning algorithms (e.g., Sarsa or Q-learning).
- State Abstraction: By allowing each module to observe only relevant state variables, Arbi-Q mitigates the "curse of dimensionality," making the learning process more efficient.
Experimental Evaluation
The approach was evaluated using the Bunny-Wolf World, a standard benchmark for MRL. In this 5x5 grid environment, a bunny agent must navigate to collect food while avoiding a predatory wolf.
To test robustness, we intentionally introduced "incomparable rewards" by multiplying the wolf-avoidance rewards by a factor of 10, simulating a scenario where modules are poorly matched and would typically break a standard MRL system.
Results and Discussion
The experimental results highlight several key advantages of the Arbi-Q approach:
- Robustness to Scale Mismatch: While traditional additive methods like GM-Sarsa suffer significant performance degradation when reward scales are mismatched, Arbi-Q shows zero degradation. It maintains the same average score per time step regardless of the internal scaling of its sub-modules.
- Algorithm Independence: Arbi-Q performs equally well whether the modules use on-policy (Sarsa) or off-policy (Q-learning) methods, a major improvement over previous techniques sensitive to the underlying algorithm.
- Convergence and Efficiency: Arbi-Q converges to an optimal policy at a rate comparable to traditional MRL. The hierarchical structure actually simplifies the arbitrator's learning task, as its action space is limited to the number of modules rather than the total number of primitive actions.
Conclusion
Arbi-Q provides a robust framework for composable reinforcement learning. By treating module selection as a learning problem, it eliminates the need for manual reward tuning and allows for the creation of reusable, "plug-and-play" RL components. Future work will explore automated state abstraction for the arbitrator and methods to allow for "compromise" actions between modules, addressing the current action bottleneck where the agent must follow a single module's advice at each step.
Summary updated