Abstract
This paper introduces Interactive Bayesian Reinforcement Learning (I-BRL), a framework designed for optimal interaction with self-interested agents using arbitrary parametric models. Traditional Bayesian Reinforcement Learning (BRL) often relies on the restrictive Flat-Dirichlet-Multinomial (FDM) prior, which assumes independence between state-action transitions and fails to generalize opponent behavior. I-BRL overcomes these limitations by allowing the use of domain-specific parametric models, enabling agents to exploit prior knowledge and generalize behavior across states. We demonstrate that I-BRL achieves superior performance and computational efficiency in complex multi-agent environments, including intersection navigation and classic game-theoretic benchmarks.
Introduction
In multi-agent reinforcement learning, an agent must balance the exploration of an opponent's strategy with the exploitation of its current knowledge. Bayesian Reinforcement Learning (BRL) provides a formal framework for this by maintaining a belief distribution over possible models of the environment. However, standard BRL implementations typically use the Flat-Dirichlet-Multinomial (FDM) prior. FDM treats every state-action pair as independent, which is unsuitable for multi-agent settings where an opponent's behavior (e.g., aggressiveness in driving) is often consistent across different states. This lack of generalization results in slow learning and an inability to incorporate expert domain knowledge.
The I-BRL Framework
The proposed I-BRL framework generalizes BRL to accommodate arbitrary parametric models and priors.
Arbitrary Modeling and Analytical Solutions
I-BRL allows practitioners to utilize any parametric model, such as traffic engineering models for autonomous driving. A key theoretical contribution is the proof that the posterior belief and optimal value function can be represented exactly using a finite set of parameters, even without conjugate priors. This breaks the common assumption that conjugacy is required for tractability.
Practical Approximation: PB-BACKUP
To address the computational complexity of planning over a continuous belief space, the authors introduce PB-BACKUP, an adaptation of Point-Based Value Iteration (PBVI). By sampling reachable beliefs and projecting the value function onto a fixed set of basis functions, I-BRL maintains polynomial-time complexity, making it feasible for real-world applications.
Experimental Evaluation
The framework was tested across three stochastic games: Intersection Navigation, Multi-Agent Chain World, and the Iterated Prisoner’s Dilemma (IPD).
The intersection navigation task, based on a 2007 DARPA Urban Challenge scenario, requires the agent to navigate a 900-state simulation while interacting with an opponent using the Gipps' car-following model.
Results and Discussion
I-BRL demonstrated significant improvements over existing methods like BPVI and Hyper-Q. In the intersection task, it halved the performance gap between a standard agent and one with perfect knowledge of the opponent.
Beyond performance, I-BRL proved more efficient in online execution. In Chain World simulations, I-BRL was nearly twice as fast as BPVI (2.5 hours vs. 4.2 hours), as it shifts the heavy computational burden to an offline planning phase. This real-time efficiency is critical for safety-critical systems like autonomous vehicles.
Conclusion
I-BRL provides a robust and flexible framework for multi-agent interaction by leveraging parametric models for better generalization. While the framework currently relies on discretized spaces and pre-defined basis functions, it sets the stage for future work in continuous state-action spaces and automated basis function selection. By moving complex reasoning to an offline phase, I-BRL enables fast, near-optimal decision-making in dynamic environments.
Summary updated