A General Framework for Interacting Bayes-Optimally with Self-Interested Agents using Arbitrary Parametric Model and Model Prior

Trong Nghia Hoang, Kian Hsiang Low

Abstract

This paper introduces Interactive Bayesian Reinforcement Learning (I-BRL), a framework designed for optimal interaction with self-interested agents using arbitrary parametric models. Traditional Bayesian Reinforcement Learning (BRL) often relies on the restrictive Flat-Dirichlet-Multinomial (FDM) prior, which assumes independence between state-action transitions and fails to generalize opponent behavior. I-BRL overcomes these limitations by allowing the use of domain-specific parametric models, enabling agents to exploit prior knowledge and generalize behavior across states. We demonstrate that I-BRL achieves superior performance and computational efficiency in complex multi-agent environments, including intersection navigation and classic game-theoretic benchmarks.

Introduction

In multi-agent reinforcement learning, an agent must balance the exploration of an opponent's strategy with the exploitation of its current knowledge. Bayesian Reinforcement Learning (BRL) provides a formal framework for this by maintaining a belief distribution over possible models of the environment. However, standard BRL implementations typically use the Flat-Dirichlet-Multinomial (FDM) prior. FDM treats every state-action pair as independent, which is unsuitable for multi-agent settings where an opponent's behavior (e.g., aggressiveness in driving) is often consistent across different states. This lack of generalization results in slow learning and an inability to incorporate expert domain knowledge.

The I-BRL Framework

The proposed I-BRL framework generalizes BRL to accommodate arbitrary parametric models and priors.

Arbitrary Modeling and Analytical Solutions

I-BRL allows practitioners to utilize any parametric model, such as traffic engineering models for autonomous driving. A key theoretical contribution is the proof that the posterior belief and optimal value function can be represented exactly using a finite set of parameters, even without conjugate priors. This breaks the common assumption that conjugacy is required for tractability.

Practical Approximation: PB-BACKUP

To address the computational complexity of planning over a continuous belief space, the authors introduce PB-BACKUP, an adaptation of Point-Based Value Iteration (PBVI). By sampling reachable beliefs and projecting the value function onto a fixed set of basis functions, I-BRL maintains polynomial-time complexity, making it feasible for real-world applications.

Experimental Evaluation

The framework was tested across three stochastic games: Intersection Navigation, Multi-Agent Chain World, and the Iterated Prisoner’s Dilemma (IPD).

Figure 2: The intersection navigation task setup, where an agent must interact with a self-interested opponent whose behavior follows a parametric traffic model.

The intersection navigation task, based on a 2007 DARPA Urban Challenge scenario, requires the agent to navigate a 900-state simulation while interacting with an opponent using the Gipps' car-following model.

Results and Discussion

I-BRL demonstrated significant improvements over existing methods like BPVI and Hyper-Q. In the intersection task, it halved the performance gap between a standard agent and one with perfect knowledge of the opponent.

Figure 4: (a) Cumulative reward R vs. planning horizon h, showing I-BRL's superior performance. (b) Planning time vs. h, demonstrating the linear scaling of the offline phase.

Beyond performance, I-BRL proved more efficient in online execution. In Chain World simulations, I-BRL was nearly twice as fast as BPVI (2.5 hours vs. 4.2 hours), as it shifts the heavy computational burden to an offline planning phase. This real-time efficiency is critical for safety-critical systems like autonomous vehicles.

Conclusion

I-BRL provides a robust and flexible framework for multi-agent interaction by leveraging parametric models for better generalization. While the framework currently relies on discretized spaces and pre-defined basis functions, it sets the stage for future work in continuous state-action spaces and automated basis function selection. By moving complex reasoning to an offline phase, I-BRL enables fast, near-optimal decision-making in dynamic environments.

Summary updated

Abstract

Recent advances in Bayesian reinforcement learning (BRL) have shown that Bayes-optimality is theoretically achievable by modeling the environment's latent dynamics using Flat-Dirichlet-Multinomial (FDM) prior. In self-interested multiagent environments, the transition dynamics are mainly controlled by the other agent's stochastic behavior for which FDM's independence and modeling assumptions do not hold. As a result, FDM does not allow the other agent's behavior to be generalized across different states nor specified using prior domain knowledge. To overcome these practical limitations of FDM, we propose a generalization of BRL to integrate the general class of parametric models and model priors, thus allowing practitioners' domain knowledge to be exploited to produce a fine-grained and compact representation of the other agent's behavior. Empirical evaluation shows that our approach outperforms existing multi-agent reinforcement learning algorithms.
BibTeX
@inproceedings{hoang2013general,
  author = {Trong Nghia Hoang and Kian Hsiang Low},
  title = {{A General Framework for Interacting Bayes-Optimally with Self-Interested Agents using Arbitrary Parametric Model and Model Prior}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2013},
  url = {https://openreview.net/forum?id=BJ-AN4fO-H},
}
Figure